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ABSTRACT 

Purposes for academic program review and diverse 
institutional approaches to college program review are considered, 
along with possible outcomes and ways to improve program review. 
Attention is directed to program review practices at 30 
representatives institutions. As background, predominant types of 
program review over the past 15 years levels are briefly addressed: 
state-review, multicampus system review, and college review. At the 
college level, major issues include: choosing programs to review, 
accommodating multiple purposes, selecting an evaluation model, 
assessing quality, using external reviewers, increasing use of 
evaluations, and assessing the impact of evaluations. Program reviews 
at most institutions draw heavily on one or more of several models: 
goal-based, responsive, decision-making, or connoisseurship. Four 
different perspectives have been offered on how quality should be 
defined: the reputational view, the resources view, the outcomes 
view, and the value-added view. Criteria for evaluating programs 
include: quality of faculty, students, curricula, support services, 
the relevance of the program to mission, student demand and demand 
for graduates, and costs. Ten tentative proposals for improving 
program reviwrf are offered that relate to purposes, processes, and 
use of results. (SW) 
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EXECUTIVE SUMMARY 



Within the last few years, the role of academic program 
review has emerged as one of the most salient issues in 
American higher education. Nestled within a context of 
accountability, program reviews have become a doiiiinant 
and controversial activity at the institutional, system, and 
state levels. Although statutory or constitutional authority 
varies greatly, higher education agencies in all 50 states 
now conduct state-level reviews; 28 of those agencies have 
authority to discontinue programs. Moreover, a majority of 
the multicampus systems have introduced program 
reviews, and over three-fourths of the nation's colleges and 
universities employ some type of program review (Barak 
1982). The heightened interest in program review can be 
traced to a widespread interest in improving program qual- 
ity and the need to respond creatively to severe financial 
constraints and to external constituencies' expectations for 
accountability. 

The literature contains a generous amount of contro- 
versy regarding the purposes, processes, and outcomes of 
program review. The intent of this monograph is to illumi- 
nate this terrain: to capture the diverse institutional 
approaches to review, to examine the central issues, and to 
reflect on ways in which program review might be 
improved. Toward that end, the report is based on a 
review of the literature and an analysis of program review 
practices at 30 representative institutions. 

What Distinguishes Current Program Reviews? 
Colleges and universities have a long-standing tradition of 
program evaluation, a tradition that can be traced from 
colonial and antebellum colleges to modern American uni- 
versities. The origins of program evaluation can be found 
in such developments as the elimination of Greek and 
Latin as entrance requirements and the introduction of 
mathematics, modern languages, and the social sciences. 
Until well into this century, program reviews were viewed 
largely as internal matters, initiated most often to reform 
and revitalize the curriculum. The idea that program 
reviews should be conducted to demonstrate accountability 
to external constituencies is a phenomenon of the twen- 
tieth century. The gradual development of regional and 
professional accrediting associations and the creation of 
statewide governing and coordinating boards are at least 



Academic Program Reviews iii 



ERIC 



partly the result of a belief that programs must be respon- 
sive to the needs and expectations of external as well as 
internal audiences. 

Especially in the last several years, program reviews 
have been designed to achieve another major objective: 
aiding those making decisions about the reallocation of 
resources and program discontinuance. Thus, a broad 
range of expectations now exists for program review in 
higher education. Program improvement, accountability to 
external constituencies, and resource reallocation axe the 
purposes cited most often. Despite this growth in expecta- 
tions, little evidence suggests that an evaluation system 
can be designed to address multiple purposes simulta- 
neously. It is especially difficult to pursue both program 
improvement and resource reallocation at the same time 
(Barak 1982; George 1982), and an institution's interests 
are served best if reviews focused on program improve- 
ment are conducted separately from those concerned with 
reallocating resources. 

What Do Formal Evaluation Models Contribute to 
Program Reviews in Higher Education? 
Program reviews at most institutions draw heavily on one 
or more of several mudels: goal-based, responsive, 
decision-making, or connoisseurship. Although these 
models are seldom explicitly identified in descriptions of 
institutional review processes, they can be inferred from 
the procedures used. 

The goal-based model has had the most influence, offer- 
ing the advantages of systematic attention to how a pro- 
gram has performed in relation to what was intended and 
of a concern for che factors contributing to success or fail- 
ure. The characteristic of the responsive model that has 
influenced program reviews in higher education is the 
attention given to program activities and effects, regardless 
of what its goals might be. The central concern of an evalu- 
ation, according to a proponent of responsive evaluation, 
ought to be the issues and concerns of those who have an 
interest in the program, not how a program has performed 
relative to its formal goal statements (Stake 1975). 

The major contribution of the decision-making model to 
program review in higher education is the explicit attempt 
to link evaluations with decision making, thus focusing the 
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evaluation and increasing the likelihood that results will be 
used. The connoisseurship model of evaluation has a long 
tradition in higher education. It relies heavily on the per- 
spectives and judgments of experts, which are valued 
because of the individual's assumed superior knowledge 
and expertise and a commonly shared value system (Gard- 
ner 1977). 

It is unlikely that an institution would want to develop an 
evaluation system that striptly conforms to any of these 
models. Their value lies in their varying perspectives on 
how evaluations should occur. The use of features from 
each model can enrich evaluations and is more likely to 
yield useful results. 

How Should Quality Be Assessed? 
The assessment of quality has generated more confusion 
and debate than any other issue for those engaged in pro- 
gram review. Pressure to define what quality means and 
what types of information should be collected has always 
existed, but interest has been heightened by the relatively 
recent emphasis on program review for resource realloca- 
tion and retrenchment. 

Four different perspectives have been offered on how 
quality should be defined: the reputational view, the 
resources view, the outcomes view, and the value-added 
view. The reputational view assumes that quality cannot be 
measured directly and is best infei.ed through the judg- 
ments of experts in the field. The resources view empha- 
sizes the human, financial, and physical assets available to 
a program. It assumes that high quality exists when 
resources like excellent students, productive and highly 
qualified faculty, and modern facilities and equipment are 
prevalent. 

The outcomes view of quality draws attention from 
resources to the quality of the product. Faculty publica- 
tions, students' accomplishments following graduation, and 
employers' satisfaction with program graduates, for exam- 
ple, are indicators used. The problem with the outcomes 
view is that the program's contribution to the success of 
graduates, for example, is not isolated. It is assumed that if 
the graduate is a success, the program is a success. 

The value-added view directs attention to what the insti- 
tution has contributed to a student's education (Astin 
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1980). The focus of the value-added view is on what a stu- 
dent has learned while enrolled. In turn, programs are 
judged on how much they add to a student's knowledge 
and personal development. The difficulty with this view of 
quality is how to isolate that contribution. 

Most institutions assess quality by adopting aspects of 
all four views. The assumption is that quality has multiple 
dimensions and, in turn, that multiple indicators should be 
used for its assessment. A large number of quantitative and 
qualitative indicators have been suggested for making such 
assessments (Clark, Hartnett, and Baird 1976; Conrad and 
Blackburn 1985). The specific indicators to be used and the 
weight to be assigned to each should vary by institution. 
Such an approach recognizes the difficulty of making quali- 
tative judgments and suggests that they will be more accu- 
rate if different dimensions are measured and different per- 
spectives solicited. 

Do Program Reviews Make a Difference? 
Perhaps the most significant issue relating to program 
review is the effect of the considerable activity at all levels 
of higher education. The assessment of impact requires 
that attention be given to the longer-term effects of deci- 
sions that are made, that is, whether a program is stronger, 
more efficient, or of higher quality. The major criterion to 
use in assessing impact is whether an evaluation makes a 
system function better (Cronbach 1977). In conducting 
such assessments, evaluators must pay attention to effects 
that are latent as well as manifest, incremental as well as 
radical, subtle as well as obvious. 

Only a few studies have analyzed impact systematically. 
The University of California (Smith 1979) and the Univer- 
sity of Iowa (Barak 1982) benefited from program reviews, 
including providing a stimulus for change and improving 
knowledge among decision makers about programs. Not all 
analyses of impact are as positive, however. A small num- 
ber of studies (Skubal 1979; Smith 1979) have focused on 
cost savings and have found that little money is saved — 
that, in fact, reviews frequently require an increased com- 
mitment. Program reviews can have negative effects — 
unwarranted anxiety, diversion of time from teaching and 
research, and unfulfilled promises and expectations 
(Seeley 1981). 
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The continued existence and growth of program review 
processes suggest that such efforts are supported and that 
the results can be beneficial. Some believe that reviews do 
not make much difference, however, and may actually 
make things worse. The studies of impact provide insuffi- 
cient evidence for deciding which view is more correct. 
Given the plethora of program reviews at all levels of 
higher education, the need to study the effects of such 
reviews more systematically is urgent. 
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FOREWORD 



Although academic program reviews have been conducted in 
one form or another for more than 50 years, only in the last 15 
have they become a major method for change or reorientation 
in the academic community. The review process is a compli- 
cated one, however, that needs to be examined carefully. 

Before academic program review is undertaken potential 
sources of problems should be identified and eliminated as 
much as possible. Three distinct aspects of program review 
can harbor problems: the stated purpose, anticipated out- 
comes, and the conflict between the two. A vaguely defined 
purpose (e.g., "programs must be reviewed regularly") is of 
little use to those conducting a review. The purpose of a 
review should be concise and clearly enunciated. Anticipated 
outcomes also must be clearly stated since they may not be 
obtainable via the chosen method of review. Problems will 
arise if the purpose and anticipated outcomes are contradic- 
tory or if too much is asked of a single review. It should be 
acknowledged, for example, that a single academic program 
review cannot at the same time improve quality and make pro- 
gram discontinuance decisions. The question to be asked is: 
What is the use of the review and how are the results to be 
applied? To be effective, a program review must be under- 
stood in terms of precise purposes, clear outcomes, and alter- 
native review methods. 

Where does the impetus initially come from to review a par- 
ticular program? Generally, faculty members have enough 
influence through department meetings, curriculum selection, 
and faculty gatherings to shape their own programs without 
resorting to a formal review process. Therefore the impetus 
for program review usually comes from an external source. 
Five distinct bodies may initiate the action leading to a pro- 
gram review: students within a program, non-program faculty, 
administrators, accrediting bodies, and state-level agencies. It 
is helpful to understand each group's focus or motive before 
charging the investigative committee. For example, students 
within a program will complain if it does not meet their expec- 
tations. They maybe concerned with the student-teacher 
ratio, the number of teaching assistants, or the connection 
between courseworic and getting a job. Other faculty members 
may request a program review if they suspect a disproportion- 
ate workload or perceive that a particular program adversely 
affects the institution's overall academic reputation. 

Administrators, faced with shrinking resources, may call for 
a cost-effectiveness review. Accrediting bodies are usually 
concerned with a consistent minimum standard of quality for 
similar programs. Finally, state-level agencies call for program 
reviews to insure that different institutions do not compete 
within a similar geographic area. It is clear that by identifying 
the primary purpose of a review and delineating the needed 
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outcome, institutions can conduct academic program reviews 
with maximum efficiency. 

It is important to establish priorities and determine the pri- 
mary purpose for a particular review. A quality control review 
aims to assess, maintain, and improve program quality. The 
goal may be open-ended and make any recommendations that 
will improve the program. If cost-effectiveness is the purpose, 
then the outcome may be to recommend ways to cut back 
current costs while minimizing the impact on program quality. 
Likewise, if the purpose of the review is to eliminate duplica- 
tion of efforts within a single system, then the desired out- 
come may be program discontinuance. Different purposes will 
produce different outcomes, just as different groups will have 
different aims in requesting a program review. Determining 
the purpose will help choose the proper review model and out- 
comes. 

There is a need to better understand the purpose behind 
academic program review, different methods of program 
.review, possible outcomes, and implementation of the results. 
The authors, Clifton Conrad, professor of higher education at 
the University of Arizona, and Richard Wilson, assistant vice- 
chancellor for academic affairs at the University of Illinois, 
address these issues by examining different approaches to 
academic program review. Since the choice of a model has a 
major impact on the results obtained, a clear understanding of 
the intended purpose and outcomes is needed before selecting 
the evaluative tool to use. The methods of program evaluation 
differ not only in the type and number of persons xequired, 
but also in the method of data collection. 

This ASHE-ERIC report, the fifth in the 1985 series, does a 
particularly fine job in giving concrete examples of institu- 
tional practices. It also shows that one area of concern should 
be the impact of a review on the faculty. Perhaps the most 
difficult part of academic program reviews is implementing the 
recommendations once they have been made. Clearly stated 
purposes and outcomes help prepare various groups for posi- 
tive changes after the report has been completed. Future areas 
cf study might include the value of conducting follow-up stud- 
ies to determine if the recommendations were implemented, 
the extent to which they were successful, and whether the 
purposes and outcomes of the review were actually achieved. 
Academic program reviews are and will continue to be a 
major tool in all levels of institutional management. 

Jonathan D. Fife 

Series Editor 

Professor and Director 

ERIC Clearinghouse on Higher Education 

The George Washington University 
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PROGRAM REVIEW IN PERSPECTIVE 



Within the last decade, the role of academic program 
review has emerged as one of the most prominent issues in 
American higher education. Many faculty, staff, adminis- 
trators, governing boards, and state higher education agen- 
cies have become deeply — sometimes passionately — 
involved in program review. To place recent developments 
in perspective, this introduction considers the antecedents 
of current review efforts, as found in a long-standing tradi- 
tion of program evaluation in American higher education, 
and discusses the growth of three major types of academic 
program review in the last 15 years. 

"Program evaluation" refers to a range of activities con- 
cerned broadly with the assessment of new or existing pro- 
grams. 4 'Program review" is a subset of program evalua- 
tion and refers exclusively to the evaluation of existing 
programs. 

Historical Antecedents 

The contemporary concern with program review is rooted 
in a tradition of program evaluation that finds its origin in 
the seventeenth century, with the development of Harvard 
College. Program evaluation at the nation's first college 
took place as early as 1642, when the first nine graduates of 
Harvard were given a public final examination by the 
external Board of Overseers. On the day of the first com- 
mencement in America, students were required to demon- 
strate their learning in the classical languages and their 
skills in disputation on both philological and philosophical 
theses. The first Harvard degrees were conferred on these 
students who "received the approbation of the overseers" 
owing to "their proficiency in the tongues of the arts" 
(Morrison 1935, pp. 256-59). The tradition of academic 
program evaluation that began at Harvard has, in various 
forms, continued to the present. 

From the seventeenth century until well into the nine- 
teenth century, control of the academic program — even 
down to the selection of textbooks — rested mostly in the 
hands of institutional lay boards. Program evaluation was 
infrequent and, in most colleges, aimed primarily at 
improving the overall academic program, such as the elimi- 
nation of Greek and Latin as entrance requirements and 
the introduction of natural philosophy (science), mathemat- 
ics, and modern languages (Conrad and Wyer 1980, p. 10). 



O demic Program Reviews 1 



ERIC 




Most institutions held firmly to the classical curriculum, 
and major changes in educational programs were consid- 
ered unnecessary, even heretical (Harcleroad 1980, p. 4). 

Despite the influence of the Yale Report of 1828, which 
defended the classical curriculum and proved a temporary 
barrier against democratic forces pressing for control of 
higher education and fundamental academic reform (Con- 
rad 1978), demands for change in the academic program 
increased as the century unfolded. Though usually infre • 
qucnt, institutional reviews of academic program offerings 
marked a growing interest in program evaluation as a vehi- 
cle for revitalization. Disenchantment with the rigid classi- 
cal curriculum of the antebellum college gradually led 
many colleges to evaluate the classical curriculum and, 
in some cases, to introduce reforms and innovations 
(Burke 1982). 

Not until after the Civil War, however, did upheaval 
occur in the academic programs of most institutions. Many 
colleges and universities began to embrace the German 
ideal of research and the American ideal of service. As a 
result, graduate and professional education was introduced 
at many institutions, and the classical curriculum began to 
crumble as the establishment of the elective system pro- 
vided a vehicle for introducing modern subjects like his- 
tory, English, and economics. These widespread changes 
seemed to symbolize an increasingly self-conscious view 
among faculty and administrators of the importance of 
evaluating both existing and proposed programs. Further- 
more, those changes reflected the influence of the strong 
administrative leadership of such university presidents as 
Andrew White at Cornell and Charles Eliot at Harvard. No 
less telling was the emergence of the professoriate as an 
important voice in academic decision making. As faculty 
grew in size and professional stature, as more specialized 
subjects were incorporated into the curriculum, and as 
institutions grew in size and complexity, control of deci- 
sions about academic programs moved increasingly away 
from institutional governing boards and toward administra- 
tive and faculty control (Harcleroad 1980, p. 9). 

The pattern of academic program evaluation established 
in the latter part of the 1800s continued well into the twen- 
tieth century. At most institutions, programs were occa- 
sionally evaluated, usually informally as administrative 
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leadership and concerned faculty joined to evaluate exist- 
ing programs and to consider changes in the formal course 
of study. An illustration of this pattern can be found in 
developments that transpired at Harvard in 1909. President 
A. Lawrence Lowell, with the support of the faculty, intro- 
duced a program in general education designed to curb 
excessive elective freedom, overspecialization, and the 
perceived lack of educational unity in the undergraduate 
curriculum (Conrad and Wyer 1980, p. 15). 

The twentieth century has witnessed the gradual devel- 
opment of external forms of academic program evaluation, 
notably accreditation by professional organizations and 
program reviews by statewide boards of higher education. 
Professional and regional accreditation, both voluntary and 
aimed at promoting minimum standards of quality, devel- 
oped into major evaluative forces during the twentieth cen- 
tury. The first of six regional accreditation bodies was 
established in 1885, and specialized accreditation in such 
fields as medicine and law took root during the first few 
decades of the new century. Especially after 1950, most 
institutions accepted voluntary accreditation: In the subse- 
quent two decades, regional associations and as many as 
27 specialized associations accredited over 2,600 institu- 
tions (Orlans 1975, pp. 16-23). 

Although six states had established a statewide govern- 
ing board for public higher education by 1900, the emer- 
gence of such boards is primarily a phenomenon of the 
twentieth century. While only voluntary forms of institu- 
tional cooperation existed in many states during the first 
half of the current century, other states began to establish 
state higher education agencies: Ten states created state- 
wide governing boards, and two created statewide coordi- 
nating agencies (Harcleroad 1980, pp. 8-9). All of these 
state agencies were given some type of responsibility for 
academic program review and approval in addition to bud- 
get approval (Millard 1976). 

While the term "academic program review" was not used 
at the time, some state boards with responsibilities for pro- 
gram review exercised that power. Several of the statewide 
boards "moved aggressively to reduce program duplication; 
in Georgia, the agency founded in the Depression year of 
1931 eliminated ten institutions" (Berdahl 1971, p. 27). 

To encourage the wise use of limited funds to meet state 
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educational needs, the major expansion of statewide gov- 
erning and coordinating boards took place between 1950 
and 1970. By 1969, 19 states had statewide governing 
boards, 27 had coordinating boards, only one had a volun- 
tary form of interinstitutional cooperation, and three had 
no formal coordination (Harcleroad 1980, p. 10). Like 
those boards established before 1950, most of the newly 
established state agencies gradually developed formal pro- 
gram review procedures to assess proposed new programs 
and to evaluate existing programs. Because program evalu- 
ation became standard practice in many states during the 
growth of the late 1950s and 1960s, evaluating new pro- 
grams rather than established ones was emphasized 
heavily. 

By the end of the 1960s, various types of program evalu- 
ation had a long history in American higher education. On 
one level, internal program review by faculty and adminis- 
trators occurred in response to pressures for changed and 
improved programs. On another level, external program 
evaluations by accrediting agencies and statewide higher 
education agencies were designed to satisfy the account- 
ability requirements of various constituencies. Especially 
with the growth and influence of external program evalua- 
tion in the 1950s and 1960s, it became clear that state and 
institutional program evaluation had become an increas- 
ingly formal, systematic process. While the foundation for 
program evaluation had been laid, however, it was not 
until the 1970s that forces largely outside the academy 
were to make program review (internal as well as external) 
a central feature of academic program planning in the 
majority of institutions and states. 

Growth of Program Review 

The growth of program review over the last 15 years has 
derived from complex, interacting societal and educational 
forces. Several factors have contributed to the expanded 
use of program review: 

• widespread interest in maintaining and improving the 
quality of higher learning, both from within and with- 
out the academy, in a period of diminishing resources 
available to higher education; 

• new academic management techniques, such as stra- 
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tcgic approaches to academic program planning and 
budgeting (Barak 1981, p. 214); 

• severe financial strains on higher education caused by 
declining enrollments, inflation and recession, 
increased costs, and overall shrinking financial sup- 
port (Dougherty 1981); 

• external demands for institutional and programmatic 
accountability from taxpayers, legislators, student 
consumers, and other constituencies; 

• demands from governing boards and campus adminis- 
trators for more effective and efficient use of limited 
resources. 

Within the context of retrenchment and accountability 
and a rising tide of concern about institutional and program 
quality, program reviews have assumed a heightened sense 
of importance at both state and institutional levels. Not 
surprisingly, since the early 1970s they have been oriented 
increasingly toward program accountability as much as 
program improvement: Existing programs have been given 
greater scrutiny with an eye toward the reallocation of 
resources, reorganizat ; on, and program discontinuance. 

In addition to the continuing presence of accreditation as 
a major vehicle for program evaluation, three major types 
of program review have been dominant in higher education 
over the past 15 years: reviews by state higher education 
agencies, multicampus system reviews, and institutional 
reviews. 

State-level reviews 

During the last 15 years, the states' involvement in aca- 
demic program review has expanded markedly, both by 
approving proposed new programs and by reviewing exist- 
ing programs (Barak and Berdahl 1978; Green 1981). While 
a few states, such as Wisconsin and Virginia, introduced 
legislative and executive audits or reviews during the 1970s 
(Berdahl 1977), most of the growth in state-level review 
has taken place among higher education agencies. 1 In gen- 
eral, responsibilities for approval of state-level programs 
have coincided with the growth of state agencies them- 
selves: As agencies have acquired broad discretionary 
powers, their responsibilities for approving programs have 
also grown. In the last decade alone, a number of agencies 
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that previously had authority only to recommend have 
been given increased regulatory responsibility. Many agen- 
cies that formerly had weak responsibility for approving 
programs now have greater authority because of expanded 
budgetary responsibilities (Barak 1982, p, 27). In most 
states, the final authority on new programs has shifted 
from institutions to state-level higher education agencies. 

By 1981, statewide agencies or governing boards in 43 
states were involved in at least some new program 
approval for public institutions: 33 had authority to 
approve programs in the public sector, and 10 could review 
programs and make recommendations. Of these 43 states, 
nine also conducted new program approval for independent 
institutions; five agencies had authority to actually approve 
programs in the independent sector, while four could con- 
duct reviews and make recommendations (Barak 1982, p. 
27). While the statutory powers, organizational structures, 
scope of authority, and approaches to evaluation vary con- 
siderably across state higher education agencies, nearly all 
of these agencies now are responsible for approving new 
programs. 

It comes as no surprise, in an era of retrenchment and 
accountability, that state higher education agencies also 
have been increasingly active in reviewing existing pro- 
grams, a responsibility that previously was reserved almost 
exclusively for the academic community (Folger 1977, p. 
viii). Within the last decade, highly publicized state-level 
reviews have taken place in New York, Washington, Flor- 
ida, and Louisiana (Barak 1977; Bogue 1980; Feasley 1980; 
Russo, Brown, and Rothweiler 1977). 

The states' involvement in program review is growing 
(Barak and Berdahl 1978; Barak and Engdahl 1980; Berve 
1975; Wilson and Miller 1980). In 1970, only a handful of 
statewide higher education agencies conducted prograr 
reviews. Today, all 50 states have some sort of reviewing 
process, although the statutory or constitutional authority 
of state agencies varies greatly, ranging from no authority 
to limited authority for reviewing certain programs at cer- 
tain levels (graduate or undergraduate) to broad authority 
to review all programs for institutions at all levels. More- 
over, some agencies represent only the public sector or a 
segment of the public sector, while a few othe *s have 
authority in the independent sector as well. A recent sur- 
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vey found that 28 state higher education agencies have the 
authority to discontinue at least some programs, even 
though this authority sometimes is limited to just one insti- 
tutional segment (Barak 1982, p. 54). Not surprisingly, a 
few state agencies exercise authority beyond their statu- 
tory or constitutional authority; others exercise less. 

In addition to differences in their authority over program 
review, organizational structure and methods of evaluation 
differ across the states. State agencies have taken two 
major approaches to evaluation. Roughly a dozen state- 
level agencies share responsibilities for review with col- 
leges and universities. In these states— Dlinois and Califor- 
nia, for example — responsibility for review rests largely 
with individual institutions, though agencies in these states 
also may conduct reviews across all institutions in a partic- 
ular discipline or cluster of disciplines. In the majority of 
states, however, state agencies assume the major responsi- 
bility for reviewing the programs of public institutions, 
relying on a combination of outside consultants and agency 
staff (Barak 1981, p. 216; Barak 1982, pp. 55-60). While 
state agencies display diversity in levels of authority over 
and methods of program evaluation, nearly all states now 
engage in some form of approval of new programs and 
review of existing programs. 

Multicampus system reviews 

Nearly all program review activity in multicampus systems 
has developed in the last 15 years. Early in the 1970s, sys- 
temwide reviews— either of proposed or of existing pro- 
grams — seldom were conducted (Lee and Bowen 1971). By 
the middle of the decade, however, multicampus reviews 
of proposed new programs were common, and existing 
graduate and professional programs were reviewed periodi- 
cally in seven of nine multicampus systems studied in one 
survey (Lee and Bowen 1975). In each of the 19 multicam- 
pus systems of higher education in the country in 1975, the 
governing board and its staff had at least some authority to 
evaluate new programs proposed by system institutions 
and/or to eliminate or otherwise to change existing pro- 
grams (Barak and Berdahl 197S). 

Today, nearly all multicampus systems conduct some 
kind of program approval. Many systems are attempting to 
upgrade the effectiveness of their evaluations of proposed 
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new programs by such measures as using outside consul- 
tants (Barak 1982, pp. 25-27). Moreover, at least half of 
the multicampus systems for both four-year institutions 
and community colleges now undertake some level of 
review of existing programs (Barak 1982, p. 49). Studies in 
this area, however, have not found a relationship between 
program review and the size of the multicampus system. 
Nor does the presence or absence of program reviews by a 
state board seem to relate to the degree of review in multi- 
campus systems. In some states, both the coordinating 
boaru and the system office review programs, while in 
other states, the system office does not conduct separate 
reviews C arak 1982, p. 51). 

Some states show considerable overlap in the legal 
authority over program review between the governing 
boards of multicampus systems and state-level coordinat- 
ing boards (Smith 1980, p. 43). While some multicampus 
systems do not conduct program reviews or else limit their 
reviews simply to monitoring state-level or institutional 
reviews, some have introduced more comprehensive system- 
level reviews in the last few years. In California, for exam- 
ple, system-ievei reviews have become second in impor- 
tance only to institutional reviews; the state coordinating 
board (the California Commission on Postsecondary Edu- 
cation) limits its role to monitoring system-level reviews. 
In short, program reviews by multicar * systems have 
become increasingly visible in recent y<*. .. 

Institutional reviews 

Most postsecondary institutions always have had proce- 
dures for approving new programs, but the procedures usu- 
ally were informal, and a new program of study often could 
be introduced with little difficulty. During the last 15 years, 
however, program approval procedures have become much 
more formal. Nearly all postsecondary institutions now 
have a program-approval process, though final approval for 
new programs in all but seven states rests in the st?te 
higher education agency. In recent years, however, the 
major expansion of academic program evaluation by insti- 
tutions has been in terms of the formal review of existing 
programs (Munitz and Wright 1980, p. 21). While concern 
about quality clearly has provided an impetus to internal 
program review, the main reasons for this growth derive 
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from institutional concerns over the effective use of scarce 
resources and from greater requirements for accountability 
from external constituencies, especially state-level higher 
education agencies (Wilson 1984, p. 143). 

In the mid-1970s, a publication of the Carnegie Council 
on Policy Studies in Higher Education predicted that insti- 
tutions would come to rely much more heavily on program 
and course evaluation (Glenny et al. 1976). This prediction 
has proven accurate. In a survey for the Nati .ial Center 
for Higher Education Management Systems, a study that 
used a stratified sample of approximately one-third of the 
nation's postsecondary institutions, 82 percent of the insti- 
tutions surveyed reported that they employed some type of 
formal program review (Barak 1982, pp. 34-35). (All of the 
survey respondents did not respond to the same definition 
of "program review"; a few institutions identified annual 
budget reviews and even regional accreditation reviews as 
program reviews.) In addition, 2 percent of the sample 
reported the use of an informal process of program review. 
Only about 16 pei cent of the responding institutions indi- 
cated that they did not conduct program reviews. 

The study also found that the majority of institutions ini- 
tiated their present program review policies after 1970, and 
more than half said their reviews were developed after 
1975. Although large research universities were more likely 
than community colleges, small independent colleges, pro- 
prietary schools, and predominantly black institutions to 
engage in internal program review, the predominant trend 
is for institutions to engage in some formal evaluation of 
existing programs. 

In summary, program review in higher education has 
surged within the last 15 years. Rooted in a long-standing 
tradition of institutional attention to program quality and 
shaped on the anvil of a period of retrenchment and 
accountability, academic program review has emerged as a 
central area of concern in higher education. Consonant 
with the external pressures acting upon higher education, 
much of the impetus for program review has come from the 
states in the form of legislative reviews or audits and 
reviews by state-level coordinating boards. Multicampus 
system reviews and institutional reviews, however, have 
emerged of late as perhaps less visible but no less dominat- 
ing a feature of higher education. Notwithstanding the 
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inevitable concern and controversy surrounding its emer- 
gence, academic program review in American higher edu- 
ca:ion has come of age. 

Focus 

The remainder of this monograph focuses on campus- 
based, or institutional, academic program reviews. In light 
of the importance now placed on the review of e.Jsting 
academic programs at the institutional level, this inquiry 
centers exclusively on such reviews. The definition of pro- 
gram review used in this monograph is adopted from Cra- 
ven (1980b, p. 434): Academic program review refers to the 
process of defining, collecting, and analyzing information 
about an existing program or noninstructional unit to 
arrive at a judgment about the continuation, modification, 
enhancement, or termination of the program or unit. 

1. "Higher education agencies" is used to refer to the ageucy (or board) 
responsible for coordinating public higher education in a state. These 
agencies, which are sometimes called "state coordinating boards," have 
varying responsibilities, most often in the areas of planning, budgeting, 
and program review. In most states, the state agency exists at a level 
above institutional boards or system boards. In states where all public 
higher education is in one system, the system board, by definition, is the 
statewide higher education agency. 
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DIVERSE APPROACHES TO PROGRAM REVIEW 



Launched on the turbulent seas of the last decade in 
response to a variety of pressures, institutional reviews of 
existing programs have left in their wakes no single pre- 
ferred way to design and implement program reviews. This 
chapter sketches the diverse approaches by examining six 
major dimensions cutting across all program review efforts: 
the purposes of program review, the selection of programs 
for review, program reviewers, models for evaluation, 
evaluative criteria, and methodological approaches. Alter- 
native approacftes and practices are identified and dis- 
cussed for each dimension. 



Purposes 

Until the last few years, the single overriding - ason for 
conducting program reviews at most institutions was to 
improve academic programs. In the late 1970s, for exam- 
ple, a report by the Committee on Internal Program 
Review of the Council of Graduate Schools stated unequi- 
vocally that "the purpose of a well-conducted review pro- 
cess is to help the program improve" (Gentile 1980, p. 7). 
At about the same time, a national survey of institutions 
found that the most commonly cited reason for conducting 
institutional program reviews was to improve programs 
(Barak 198?, p. 39). 

The literature abounds with case studies of institutions in 
which assessment for improvement serves as the driving pur- 
pose behind program review. For example, program 
improvement is the major purpose of program review at Ohio 
State University, the University of Nebraska-Lincoln, the 
University of Illinois, the University of Iowa, and the Metro- 
politan Community College District in Kansas City, Missouri 
(Arns and Poland 1980; Barak 1982; DiBiaso and Ecker 1982; 
Seagren and Bean 1981; Wilson 1984). 

In addition to improving programs, other compelling rea- 
sons for conducting program reviews have surfaced in 
recent years: to meet state-level mandates for reviews; to 
demonstrate institutional responsiveness to constituencies, 
such as sponsoring denominations or community groups; 
to provide a foundation for allocating and reallocating 
resources; to provide information to decision makers con- 
sidering program discontinuance; and, in the case of two- 
year institutions, to respond to federal requirements for the 
evaluation of occupational programs. Usually, several of 
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these reasons provide the impetus for initiating program 
reviews at an institution. For many institutions, adding an 
additional statement of purpose has been a relatively pain- 
less way to build consensus for program review (Wilson 
1984, p. 144). As a result, institutions display diversity in 
terms of the various constellations of purposes that inform 
their program review efforts. 
| | Notwithstanding this diversity, the m^jor differences 

across institutional program reviews pertain to their pri- 
mary purposes. On the one hand, many institutions empha- 
size program improvement as the major purpose of pro- 
gram review. On the other hand, a growing number of 
institutions have embraced the reallocation of resources 
(including, in some cases, program reduction) as the driv- 
ing force behind program review. And a large number of 
institutions have embraced both program improvement and 
resource reallocation as the two driving purposes in con- 
ducting program reviews. 

Given the context of retrenchment and accountability 
confronting many postsecondary institutions, it is hardly 
surprising that the central purpose of program review in 
many institutions is driven by a desire to allocate and real- 
locate resources on a differential rather than an across-the- 
board basis. Next to program improvement, in one study, 
resource reallocation was the most frequently cited reason 
for conducting program reviews (Barak 1982, p. 39). Such 
institutions as the University of Vermont, Michigan State 
University, the University of Houston, and the University 
of Minnesota have connected program reviews to resource 
allocation (Arns and Poland 1980; Clugston i984; Munitz 
and Wright 1980). 

Perhaps the most telling feature of recent program 
reviews is that many institutions now view program review 
as a major vehicle for allocating and reallocating resources, 
thereby accomplishing more with existing resources. At 
the University of Louisville, for example, program reviews 
are examined by all of the vice presidents as well as by a 
planning staff that includes representatives from the plan- 
ning office, budget office> and institutional research office. 
Thus, program review is linked closely with planning and 
budgeting, as indicated in two of the university's major 
objectives for program review: 
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/ . To provide a basis for recommendations regarding 

a. internal allocations in the preparation of annual 
operating budgets and 

b. reallocations (in the form of budget adjustments) 
during the operating year and 

2. To seive as a building block in an evolving planning 
process that will make planning and resource alloca- 
tion increasingly rational processes (Barak 1982, p 
38). 

Some institutions faced with declining resources have 
gone so far as to use "program discontinuance"— includ- 
ing merging programs, eliminating curricula within depart- 
ments, and closing entire departments, colleges, or nonin- 
structional units— as a tool for reducing and reallocating 
resources. For those institutions, a major purpose of pro- 
gram review is to provide a foundation for reducing the 
scope of institutional program offerings so as to "extract 
programs not meeting [program priorities] ... to preserve 
the vitality of those at the heart of the intellectual enter- 
prise" (Shirley and Volkwein 1978, p. 486). The resources 
saved are reallocated to existing programs or used to sup- 
port new programs. 

No exact figures are available regarding the number of 
institutions that have made program discontinuance a 
major consideration in program review, but more and more 
institutions are moving in that direction. In response to 
state-level mandates or declining resources, several public 
institutions in the early to mid-1970s (the State University 
of New York at Albany, for example) undertook reviews 
that led to one or more programs' being discontinued. Oth- 
ers whose reviews during the last decade have led to pro- 
grams' being cut include the University of California- 
Berkeley, the University of California-Riverside, the Uni- 
versity of Pennsylvania. Florida State University, the Uni- 
versity of Wisconsin-Eau Claire, and the University of 
Michigan (Dougherty 1979, 1981). 

While most institutions involved in program review seem 
to emphasize either program improvement or resource 
reallocation (including program discontinuance), this exam- 
ination of institutional reviews suggests thai a substantial 
number cf institutions now combine both emphases as the 
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two driving purposes guiding program reviews. For exam- 
ple, until it recently placed a moratorium on program 
reviews, the University of Michigan conducted reviews 
aimed both at improving programs and reallocating 
resources, with the latter including program reduction 
(Dougherty 1981; Munitz and Wright 1980). Similarly, pro- 
gram review at the University of Tennessee at Knoxville 
emphasizes primarily these two purposes, as indicated in 
the institutional statement of review purposes: 

• To improve the quality of the University's academic 
offerings. 

• To achieve the best use of resources. 

• To foster cooperation among the academic and 
administrative units. 

• To evaluate quality, productivity, need, and demand 
within the university, state, and region. 

• To determine effectiveness and consider possible mod- 
ifications. 

• To facilitate academic planning and budgeting (Office 
of the Provost 1984). 



Program Selection 

Although it has been addressed infrequently in the litera- 
ture, the selection of programs to review is a critical deci- 
sion. Colleges and universities involved in program review 
show diversity both in regard to whether programs are 
reviewed regularly or on an ad hoc basis and in terms 
of the particular mechanisms used to select programs 
for review. 

Multicampus systems or state-level agencies now require 
public institutions in many states to review periodically all 
academic programs (graduate or undergraduate) and, in 
some cases, noninstructional units as well. Along with 
independent colleges and universities, these institutions 
have established a cycle of reviews in which all programs 
are reviewed regularly. At the State University of New 
Yo/k at Albany, for example, all departments are reviewed 
on a seven-year cycle (Dougherty 1979); Florida State Uni- 
versity reviews all graduate programs on a five-year cycle. 

In sharp contrast, some institutions conduct no regular 
reviews of academic programs but target programs for 
review on an ad hoc basis according to perceived need. Of 
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1 1 public universities that had conducted program reviews, 
four selected programs on an ad hoc or noncyclical basis 
(Dougherty 1979). 

A few institutions combine regular reviews with in- 
depth, ad hoc reviews. For example, the University of Illi- 
nois at Urbana/Champaign has a two-stage program review 
process in which all units are reviewed periodically as part 
of the first stage of review. If problems are identified, a 
second stage, or in-depth, ad hoc review is conducted (Wil- 
son 1984, p. 150). Thus, this two-stage process combines a 
regular review cycle with an additional ad hoc review 
where serious problems exist. 

To avoid superficiality, most institut&nsanvolved in pro- 
gram review select only a limited number oSTprograms for 
intensive review at any single time. This pattern holds true 
whether institutions rely on periodic or ad hoc reviews (or 
a combination of the two), although institutions with a reg- 
ular cycle of reviews conduct many more reviews than do 
insiitutions employing ad hoc reviews only. Four m^jor 
mechanisms are used to identify programs for intensive 
review: (1) logical clusters of programs or explicit criteria; 
(2) quantitative indicators; (3) the appointment of a new 
dean or department chair; and (4) as issues arise. 

Institutions that review programs regularly normally 
select programs according to logical groupings (for exam- 
ple, all programs in the social sciences) or on the basis of 
a priori criteria. An example of the former is the University 
of California-Berkeley, which reviews six to eight gradu- 
ate programs each year on the basis of logical clusters 
(Dougherty 1979). An example of the latter is the Univer- 
sity of Nebraska-Lincoln, where 20 percent of all pro- 
grams are selected for review each year on the basis of the 
following criteria: 
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Relationship of the program to other programs under 
review; marked change in student demand; planned pro- 
> gram changes; accreditation cycles; and, elapsed time 
since last major budget review of budget, staffing, or 
program for any purpose (Seagren and Bean 1981 , p. 9). 

For institutions conducting ad hoc reviews, some flag- 
ging mechanism is needed to trigger the selection of pro- 
grams for review. The most typical approach is to use 
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quantitative indicators such as cost effectiveness, degree 
productivity, student credit hours generated, faculty work 
loads, and various indicators of program quality. (Some 
institutions relying on the cyclical pattern employ quantita- 
tive indicators to begin their cycles.) In most institution, 
only negative indicators trigger the review mechanism. A 
few institutions select one or two programs of high quality 
to review along with programs suspected of being rela- 
tively low in quality or having other negative indicators 
(Dougherty 1979). Institutions using quantitative indicators 
include the University of Wisconsin-Madison and the Uni- 
versity of Wisconsin-Eau Claire (Dougherty 1979). 

Although it is not a common practice, a few institutions 
review programs in conjunction with the appointment of a 
new dean or department chair. The rationale for this 
approach is that it provides valuable background informa- 
tion to the search committee and to the new appointee. A 
new chair or department head will know where to devote 
initial attention if a program review has been completed. 
Institutions using this approach include the University of 
Michigan and the University of Pennsylvania. 

A growing number of institutions select programs for 
review as issues arise, such as when enrollment drops for a 
program, when a program receives a negative accieditation 
report, or when concerns are raised about the quality of 
the program. At the University of California-Berkeley, for 
example, the School of Criminology was reviewed because 
questions were raised about the quality of the program. 
Following a review of the school, which focused solely on 
qualitative considerations, the program was discontinued 
(Dougherty 1979). At the University of Michigan, the with- 
drawal of outside funds from the Population Planning Pro- 
gram, coupled with the request of the department for 
increased support, led to a review that resulted in the pro- 
gram's discontinuance. 

In many colleges and universities, concerns about pro- 
gram quality and student demand have made certain fields 
especially vulnerable to ad hoc reviews. Education, for 
example, has become a popular target on many campuses, 
including the University of Kansas and the University of 
Arizona. Similarly, social work, library science, and 
humanities disciplines are being revtewed more frequently 
than other programs. 
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Program Reviewers 

A critical dimension in any review process is the selection 
of evaluators to conduct the reviews. The process of selec- 
tion is affected by the purposes of the review and the insti- 
tutional style of governance. In many institutions, the 
responsibilities for review are assigned by a central review 
group or major administrator, often in consultation with a 
faculty senate and individuals representing the program 
under review. In other institutions, the central review 
group or administrator overseeing the review sets forth 
broad guidelines regarding the type(s) of reviewers to be 
used (for example, external evaluators), leaving the choice 
of evaluators to the program under review. In a few institu- 
tions, reviewers are selected jointly by a central review 
committee or administrator (or administrative group) and 
the program under review (Mims 1978). 

Regardless of who selects the program reviewers, vir- 
tually all program reviews in higher education are estab- 
lished on the premise that program evaluation should be 
based on professional judgment. Notwithstanding this com- 
mon feature, however, three general approaches to evalua- 
tion affect the selection of program reviewers: internal 
review, external review, and several combinations of inter- 
nal and external review. Table 1 displays these alternative 
approaches, 

Self-review or internal review probably is the dominant 
approach to evaluation in higher education. The tradition 
of internal self-evaluation is founded on the belief that pro- 
gram faculty best know the strengths and limitations of the 
program, and many institutions continue to make self- 
review the cornerstone of program review. Especially 
when the major purpose of program review is to improve 
the program, colleges and universities operate on the 
assumption that departmental or program faculty and in 
some instances students should be the primary reviewers. 
In the last few years, however, alternative and supplemen- 
tary approaches to program review have become preva- 
lent. Many institutions have chosen to give a much larger 
role to external program reviewers, that is, individuals not 
associated with the program under review. 

External program review has begun to play a prominent 
role throughout higher education for a number of reasons. 
In many public colleges and universities, mandates for 
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TABLE 1 

THREE APPROACHES TO SELECTING 
PROGRAM REVIEWERS 

Internal or Self-Review (internal reviewers) 

• Program faculty 

• Students in program (current and/or former) 

External Review (external reviewers) 
Within Institution 

• Faculty peers (outside discipline or field being evaluated) 

• Professional staff 
Outside Institution 

• Faculty peers (usually in discipline or field being evaluated) 

• Professional evaluators 

Internal/External Review (internal/external reviewers) 

• Mixed review group 

• Multiple reviews 

Source: Adapted from Mims 1978, p. 11. 

state-level and multicampus review require that external 
reviewers be an integral part of the process. No less teli- ? 
ing, the context in which reviews are taking place has 
changed dramatically in the last few years in public and 
independent institutions alike. The principal purpose of 
program review has shifted increasingly away from pro- 
gram improvement to the reallocation of resources and , 
even program discontinuance. As a result, many colleges 
and universities have turned to external reviews to 
enhance the objectivity and credibility of a review process 
that is intended to provide a foundation for making difficult 
decisions about the future of programs. 

In an external review, program reviewers may be chosen 
from within the institution but external to the program 
under review or from outside the institution. In the former 
case, reviewers are usually faculty peers from outside the 
discipline or professional field being evaluated, though 
administrators and even students may also be involved. In 
a few institutions, professional staff who are experts on 
program evaluation participate in the review, but in most 
cases these individuals serve only as advisors (Wood and 
Davis 1978). 

An increasingly common approach to external program 
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review is to employ review consultants from outside the 
college or university. In a small number of institutions, 
outside consultants with general backgrounds in program 
evaluation are hired to conduct the review. In most 
instances, however, professional evaluators are employed 
only to assist in designing the review process itself rather 
than to review programs. The more typical approach, 
which can be quite expensive, is to choose several faculty 
peers from other institutions who are in the same discipline 
or field as the program under review. One study of pro- 
gram review practices found that roughly half the institu- 
tions conducting reviews used faculty from other institu- 
tions as program reviewers (Barak 1982, p. 40). 

To balance the various strengths and weaknesses of self- 
review and external review, a large and growing number of 
institutions combine internal and external review. In some 
institutions, a single review team — a mixed review group — 
is selected consisting of both internal and external review- 
ers. The external reviewers may include faculty from 
within or without the institution or both. The University of 
Illinois at Urbana/Champaign, for example, uses a mixed 
review group that includes program faculty and university 
faculty from outside the program being reviewed to review 
its programs. 

A more common approach to internal/external program 
review is to use multiple reviews in which self-reviews and 
external reviews are conducted separately. At Trenton 
State College, two separate reviews are conducted — one 
self-review and one external review (the latter conducted 
by peer faculty from other institutions who are in the disci- 
pline or field being reviewed). In some institutions, three 
separate reviews are conducted. At the University of Ari- 
zona College of Education during the 1983-84 academic 
year, for example, a self-review by faculty in the college 
was followed by a review by a universitywide committee 
comprised of faculty from outside the college. Subse- 
quently, a committee comprised of faculty in education 
from other universities conducted an external review. 

Models for Evaluation 

Along with the selection of program reviewers, the models 
used to evaluate programs are a critical aspect of program 
review. An evaluation model not only provides the overall 



*ndemic Program Reviews 



19 



framework for evaluation but also gives shape to the 
research questions, organizes and focuses the evaluation, 
and informs the process of inquiry. 

The models used in program review are seldom made 
explicit, but all institutions use some model or combination 
of models to guide their review efforts. These models are 
reflected in the guidelines developed by the central review 
committee or administrative group designing the review 
process and in the reviews themselves. Thus, evaluation 
models can be inferred from the guidelines and reports 
accompanying program reviews as well as from the litera- 
ture on program review. 

Based on their program review guidelines and practices, 
most institutions use various models for program evalua- 
tion, emphasizing one of four evaluation models: a goal- 
based model, a responsive model, a decision-making 
model, or a connoisseurship model. As displayed in table 
2, this taxonomy draws heavily on the classifications of 
program evaluation models of several writers (Gardner 
1977; Gubaand Lincoln 1981; House 1978; Mines, Gres- 
sard, and Daniels 1982; Popham 1975). These models are 
discussed in the follo wing paragraphs in terms of their 
focus and organizing principle, evaluation design, and 
application in an institutional setting. 




God-based model 

Grounded in the work of Tyler (1949), the goal-based 
approach to evaluation is the oldest and most widely used 
model of evaluation in higher education. Several alterna- 
tive goal-based models have been introduced in the last 
several decades, but all goal-based evaluation is organized 
around the extent to which the program under review 
achieves its intended objectives (Craven 1980b, p. 436). 



A goal-based model defines evaluation as the process of 
identifying program goals, objectives, and standards of per- 
formance, using various tools to measure performance, and 
comparing the data collected against the identified objec- 
tives and standard? to determine the degree of congruence 
or discrepancy (Gardner 1977, pp. 577-78) In this model, 
the most important components of the evaluation design 
are the identified goals, objectives, and criteria used to 
judge relative success or failure. Decisions about measure- 
ment and interpretation are normal* * left in the hands of 



TABLE 2 

A TAXONOMY OF EVALUATION MODELS 



Model 

Goal-based 
Model 



Responsive Model 



Decision-making 
Model 



Proponents* 

Tyler 1949 (Behavioral Objectives) 
Provus 197) (Discrepancy Model) 
Popham 1975 

Scriven 1973 (Goal-free Model) 
Stake 1975 (Responsive Model) 
Parlett and Deardon 1977 (Illuminative 

Evaluation Model) 
Guba and Lincoln 1981 (Naturalistic 

Responsive Model) 

Stufflebeam et al. 1971 (Context- 
Input-Process-Product Model) 

Alkin 1972 (UCLA Center for the 
Study of Evaluation Model) 

Alkin and Fitz-Gibbon 1975 



Connoisseurship 
Model 



Mode! Organizer 
Goals and objectives 



Concerns and issues 
of stakeholders 



Decision making 



Critical review by 
connoisseurs 



Eisner 1975 (Connoisseurship Model) 
Evaluation models associated with particular individuals are indicated in parentheses. 



Evaluation Quef tions 

To what extent is the program 
achieving its objectives? 



What are the activities and effects of 
the program? What does the program 
look like from a variety of 
perspectives? 



To what extent is the program 
effective? In light of alternative 
decisions, what is the worth of the 
program? 



How do critics interpret and evaluate 
the programs? 
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those individuals actually conducting the evaluation, 
although final judgments about merit and worth of the pro- 
gram usually are made by those overseeing the evaluation. 

While all goal-based models place m^jor emphasis on 
assessing the degree of congruence between objectives and 
performance, a goal-based approach need not focus exclu- 
sively on the extent to which goals and objectives are met. 
In the Provus (1971) model, for example, attention also is 
placed on describing and interpreting program performance 
and on exploring the reasons for relative success or failure. 
Thus, a goal-based process can have a formative role (eval- 
uation aimed at improving a program) as well as a summa- 
tive one (evaluation of a program for purposes of making a 
decision about the program) (Worthen and Sanders 1973, 
pp. 63-65). Still, as employed in academic program 
reviews, most goal-based evaluations seem to be predomi- 
nantly summative: Designing an evaluation that will pro- 
vide a foundation for making decisions about a program 
(resource allocation and program continuance, for exam- 
ple) is emphasized. 

The design of goal-based evaluation may vary from sim- 
ple to complex but invariably includes the following ele- 
ments: 

• clarification of the gosh and objectives of the program 
under review 

• identification of the factors oi variables affecting per- 
formance 

• delineation of the criteria and standards against which 
program performance will be assessed 

• development of techniques and procedures for collect- 
ing data on performance 

• data collection 

• comparison of the data with the previously identified 
criteria and standards, leading to a judgment of worth 

• communication of the findings (Gardner 1977, p. 578). 

As the major approach to evaluation in higher education, 
the goal-based model drives academic program review in 
hundreds of colleges and universities. A good example of 
the approach is the evaluation system used at Broome 
Community College in Binghamton, New York. Adapted 
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from the model of program evaluation for two-year col- 
leges called Reality-Based Evaluation (RBE) for Two- Year 
Occupational Programs (developed by the Cornell Institute 
for Research and Development in Occupational Educa- 
tion), this model aims to improve decision making in areas 
such as goals, standards, and program expansion or curtail- 
ment. As implemented at Broome Community College, the 
RBE model is a prototypical example of the goal-based 
approach. The evaluation model encompasses three gen- 
eral phases. The first phase attempts to clarify program 
objectives and activities, the second phase involves gather- 
ing information from a variety of sources to examine the 
extent to which program objectives are being met, and the 
third phase focuses on data analysis and overall interpreta- 
tion of the findings. A written report describes all three 
phases of ths review and presents recommendations 
(Barak 1982, pp. 105-7). 

Responsive model 

In the last few years, a sharply contrasting approach to the 
goal-based model has been developed that has had consid- 
erable influence on program review practices at many insti- 
tutions. The responsive model finds its roots in the goal- 
free model of Scriven (1972, 1973), although it has gone 
beyond his formulation. Briefly, Scriven took issue with 
the widely accepted notion that goals and objectives should 
drive the evaluation. He argued that "unintended effects" 
or 44 side effects" are often as important as intended 
effects. Scriven's goal-free model of evaluation was aimed 
at judging the effects of programs independent of what the 
effects were intended to be. In essence, Scriven's orga- 
nizer for evaluation became "effects" rather than goals 
and objectives (Guba and Lincoln 1981, p. 17). 

Building upon Scriven's contention that program goals 
and objectives should not drive progiam evaluation, Stake 
(1975) developed the responsive model; a number of other 
writers (Guba and Lincoln 1981 ; Parlett and Deardon 1977) 
supported and further developed this approach. In this 
model, evaluation focuses more on program activities than 
on the program's stated goals and objectives and is orga- 
nized around the 4 'concerns and issues of stakeholding 
audiences" (Guba and Lincoln 1981, p. 23). 



As the major 
approach to 
evaluation in 
higher 

education, the 
goal-based 
model drives 
academic 
program 
review in 
hundreds of 
colleges and 
universities. 
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To emphasize evaluation issues that are important fcr 
each particular program, I recommend the responsive 
evaluation approach. It is an approach that trades off 
some measurement precision in order to increase the 
usefulness of the findings to persons in and around the 
program. ... An educational evaluation is responsive 
evaluation if it orients more directly to program activi- 
ties than to program intents; responds to audience 
requirements for information; and if the different value 
perspectives present are referred to in reporting the suc- 
cess and failure of the program (Stake 1975, p. 14). 

Thus defined, responsive evaluation can serve many differ- 
ent purposes: to serve as a tool for decision making, to 
improve understanding, to facilitate program improvement, 
and so on. The needs met by a given evaluation, however, 
are determined by the "different purposes and information 
needs of different audiences" (Stake 1975, p. 15). In pro- 
gram review, for example, the evaluation is shaped by con- 
cerns expressed by different audiences, such as administra- 
tors, students, program faculty and staff, and faculty from 
outside the program. 

Responsive evaluation, then, is a process of collecting, 
analyzing, and interpreting information about a program in 
light of the concerns and issues of audiences that have a 
stake in the evaluation. Accordingly, program goals and 
objectives may or may not be centrally important to the 
issues identified or to the strengths and weaknesses of the 
program relative to those issues. Instead, all aspects of the 
program under review are taken into consideration relative 
to audience-defined issues, and "no single element 
(whether goals, resources, processes, or participants) is 
preconceived as being necessarily more important to the 
evaluation than another" (Gardner 1977, p. 584). 

A responsive evaluation cannot be designed fully before 
the actual evaluation, however, inasmuch as each step is 
determined in part by its predecessors (Guba and Lincoln 
1981 , p. 36). These steps, which should not be carried out 
serially, include: 

1. interviewing clients, program staff, and concerned 
audiences 

2. limiting program scope 
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3. surveying program activities 

4. discovering stated and real purposes of the program 
and the concerns of various audiences 

5. conceptualizing issues and problems 

6. identifying data needs based on identified issues 

7. selecting observers and judges 

8. collecting data 

9. preparing portrayals and case studies based on 
"themes" 

10. matching issues to audiences 

11. presenting findings (Stake 1975). 

To be sure, few (if any) colleges and universities have 
systematically applied the responsive model in their pro- 
gram reviews, but this model clearly has appeal to institu- 
tions that believe traditional approaches — especially the 
goal-based model— suffer limitations. As a result, some 
institutions have combined various aspects of the respon- 
sive model with other approaches to evaluation. The aca- 
demic review process at the University of Colorado-Boul- 
der, for example, is coordinated by an Academic Program 
Review Committee and consists of three stages: (1) a self- 
study conducted by the program under review; (2) an inter- 
nal review by a committee of Boulder faculty and students 
from fields outside the program under review; and (3) an 
external review by disciplinary experts. In a manner con- 
sistent with the responsive approach to evaluation, the per- 
spectives of several constituencies are sought. For exam- 
ple, the self-study committee consults the college dean 
early in the review process, students are surveyed during 
the self-study process, and faculty views are solicited dur- 
ing the preparation of the self-study report. Further, the 
internal review team interviews a representative group of 
faculty members and students, and the external review 
committee does extensive interviewing (University of Col- 
orado-Boulder n.d.). The solicitation of views from all 
those constituencies connected to a program is characteris- 
tic of most evaluation processes in higher education and is 
a central tenet of the responsive model 

Decision-making model 

Within the last several years, especially at institutions 
where academic program reviews are conducted to reallo- 
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cate resources or to decide whether a program should be 
continued, considerable interest has been evinced in using 
the decision as the m^jor organizer for evaluation. In the 
decision-making model, neither goals and objectives (the 
goal-based model) nor the concerns and issues of various 
stakeholders (the responsive model) are necessarily a cen- 
tral component in the design of the evaluation. 

Two prototypical decision-making models are the Context- 
Input-Process-Product (CIPP) model (Stufflebeam et al. 1971) 
and the UCLA Center for the Study of Evaluation (CSE) 
model (Alkin 1972). Both models are nearly identical in their 
m^jor characteristics: The CIPP model defines evaluation as 
"the process of delineating, obtaining, and providing useful 
information forjudging decision alternatives'* (Stufflebeam et 
al. 1971 , p. 40), whereas the CSE model defines it as "the 
process of ascertaining the decision areas of concern, select- 
ing appropriate information, and collecting and analyzing 
information in order to report summary data useful to deci- 
sion makers in selecting among alternatives" (Alkin 1972, p. 
12). 

Evaluation in the decision-making model is conducted 
for purposes of decision making and accountability; the 
CIPP model is built on the premise that information must 
be useful to decision makers. Evaluation is viewed as a 
cyclical, continuing process in which a formal "feedback 
mechanism provides for continuing assessment of decision- 
information needs and the obtaining and providing of infor- 
mation to meet those needs" (Gardner 1977, pp. 580-81). 

A central feature of the CIPP model is that different 
types of decisions require varying kinds of information and 
hence different types of evaluation activities. Four types of 
evaluation have been identified, each corresponding to a 
different type of decision: (1) context evaluation, which 
assists decision makers in determining goals and objec- 
tives; (2) input evaluation, which assists decision makers in 
clarifying alternative ways of achieving program goals and 
objectives; (3) process evaluation, which provides feed- 
back to decision makers; and (4) product evaluation, which 
provides decision makers with information as to whether a 
program should be continued, modified, or terminated 
(Stufflebeam et al. 1971 , pp. 80-84). In the CIPP model, 
only context evaluation is an ongoing process. Input, pro- 
cess, and product evaluations are initiated only when the 



ERLC 



41 



context evaluation points to a problem, need, or opportu- 
nity (Guba and Lincoln 1981 , p. 15). 

The design is the same for ail four types of evaluation 
encompassed in the CIPP model, and it has three separate 
elements: delineating, obtaining, and applying. Delineating 
involves focusing the evaluation by identifying the decision 
situations to be served and defining the criteria to be used 
by decision makers in assessing alternatives, obtaining sim- 
ply refers to collecting and processing the pertinent infor- 
mation, and applying refers to analyzing the data and 
reporting the findings to decision makers (Worthen and 
Sanders 1973, p. 144). fa short, evaluation in the CIPP 
model is a process of delineating, obtaining, and applying 
information to the needs of decision makers. 

While the CIPP model has not been widely employed in 
higher education, a growing number of institutions have 
initiated decision-oriented program evaluations. These 
evaluations are structured to meet administrative informa- 
tion needs by providing relevant and timely information. A 
good case in point is the University of Houston central 
campus, which has implemented a systematic, decision- 
oriented program evaluation system aimed at providing a 
foundation for allocating and reallocating funds among aca- 
demic departments. While departmental resource needs 
("relative resource consumptiveness") and program cen- 
trality are considerations in this model, the most essential 
element is ongoing qualitative evaluation of departments 
and programs. Through a complex series of matrices, the 
University of Houston has designed a system in which rel- 
evant information about academic programs is provided to 
campus administrators to facilitate decision making about 
the distribution of campus resources (Munitz and Wright 
1980, pp. 31-38). 

Connoisseurship model 

In contrast to the first three models, Eisner's connoisseur- 
ship model (1975) represents a strikingly different approach 
to evaluation — one that a substantial number of institutions 
have used, at least in part, to guide academic program 
reviews. At first glance, this model may appear orthogonal 
to the other three: It also necessarily emphasizes the rcle 
of the program reviewer or evaluator. Yet in several impor- 
tant respects the model differs markedly from the other 
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models. First, and most important, the catalyst for evalua- 
tion is not the constellation of goals and objectives, not the 
concerns and issues of stakeholders, not decision making, 
but the connoisseur. Second, the model uses the human 
being (connoisseur) as the primary instrument of measure- 
ment (to some extent, also true with the responsive 
model). In turn, data collection, analysis, and interpreta- 
tion are guided primarily by the connoisseur rather than 
the evaluation design and are less open to direct inspection 
than in the other models. Third, the connoisseurship model 
is based more on a metaphoric approach to evaluation 
(Guba and Lincoln 1981, p. 19). 

As defined by Eisner, a connoisseurship model is based 
on two concepts drawn from the domain of art criticism: 
educational connoisseurship and educational criticism. 
Like works of art, education is highly complex, requiring 
4 'connoisseurship," or the "art of perception that makes 
such complexity possible" (Eisner 1975, p. 1). Because of 
his training and background, the connoisseur is by defini- 
tion the individual best able to appreciate the subtleties and 
nuances of what he encounters. 

// connoisseurship is the art of appreciation, criticism is 
the art of disclosure. What the critic aims at is not only 
to discern that character and qualities constituting the 
object or event [but also to provide] a rendering in lin- 
guistic terms of what it is he or she has encountered so 
that others not possessing his level of connoisseurship 
can also enter into the work. . . . The function of criti- 
cism is educational. Its aim is to lift the veils that keep 
the eyes from seeing by providing the bridge needed by 
others to experience the qualities and relationships 
within some area of activity. . . . The critic must talk or 
write about what he has encountered; he must . . . pro- 
vide a rendering of the qualities that constitute that 
work, its significance, and the quality of his experiences 
when he interacts with it (Eisner 1975, p. 1). 

In a connoisseurship model, the evaluation is structured 
in accordance with the expectations of those served by the 
evaluation. In turn, the connoisseur may well bo expected 
to consider guidelines and criteria. The evaluation, how- 
ever, is structured basically around interaction between 
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the connoisseur and the program under review, with few 
a priori constraints. As a result, the standards and criteria 
that connoisseurs use in reaching their judgments derive 
primarily from their experience as professionals and upon 
the collective experience of the profession (House 1978). 
In short, the judgment of the professional is accepted on 
the basis of both that individual's assumed superior knowl- 
edge and expertise, which accompanies his stature in the 
particular field in question, and a commonly shared value 
system (Gardner 1977, p. 574). 

In the connoisseur model, then, the connoisseur alone 
guides the evaluation. Operating with few constraints, the 
evaluator is initially a processor of information, collecting 
data in whatever way he prefers — for example, documents, 
interviews, or observation. At the same time, however, he 
is constantly judging the program under review — sifting 
through the evidence, weighing relevant data, seeking to 
understand the "character" and "qualities" of the pro- 
gram, and fleshing out deeper meaning — in much the same 
way that the art critic judges a work of art. At the point of 
closure, the values and criteria that inform his judgments 
may or may not be fully disclosed, but whether left implicit 
or made explicit, the final report is based on the subjective 
judgments emanating from the connoisseur's own thought 
processes (Gardner 1977, p. 575), on his own construction 
of social reality. 

One institution integrating the connoisseurship model 
into its overall approach to academic program review is the 
University of Nebraska-Lincoln (UN-L). At UN-L, the 
Academic Planning Committee, a faculty-dominated, cam- 
puswide committee, oversees the evaluation of all instruc- 
tional and non instructional programs on a five-year cycle. 
While a goal-based approach to self-study marks the initial 
step in the design of the evaluation, the key component of 
the review is the visit of the review team (Seagren and 
Bean 1981, p. 22). The typical review team includes a stu- 
dent from the department, a facilitator from the universi- 
ty's Office of Program Review, a UN-L faculty member 
from outside the department under review, and two ex- 
ternal members (one to serve as chair) of the discipline 
under review. 

While its review team is not entirely comprised of "con- 
noisseurs," UN-L's approach is consonant with the con- 
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noisseurship model. Not only is the review team made up 
primarily of professionals-qua-connoisseurs; it is also given 
few guidelines and criteria that might constrain its efforts 
to explore the status and function of the program within 
UN-L and to assess program quality. Moreover, in lan- 
guage highly compatible with the connoisseurship model, 
the review team is encouraged to provide a "vision of both 
what the department is, or seems to be, and what it could 
be, given its qualities and the constraints [that] will affect 
its future development" (Seagren and Bean 1981, p. 25). 
With the review team as the central component in the 
review process, UN-L clearly has integrated several ele- 
ments of the connoisseurship model into its program 
review process. 



Evaluative Criteria 

Nearly all colleges and universities involved in academic 
program review use evaluative criteria to some degree. To 
be sure, some institutions employ models, such as the con- 
noisseurship or responsive approaches, that in theory 
reject the use of a priori criteria. In practice, however, vir- 
tually all institutions predetermine criteria to guide their 
program reviews. 

Most institutions use similar evaluative criteria. While 
institutions may use different nomenclature and only infre- 
quently use exactly the same criteria, an "essential same- 
ness" pervades the general criteria used in reviews. Table 
3 shows the most common evaluative criteria identified in 
the liteiature (Barak 1982; Blackburn and Lingenfelter 
1973; Melchiori 1980; Shirley and Volkwein 1978) and in 
the sample of institutions used for this monograph. 
Although all 'nstitutions do not use all six of the subcriteria 
listed under "quality" in table 3, we know of no institution 
involved in program review that does not identify quality 
as a central criterion for evaluation. Furthermore, a major- 
ity of institutions use criteria related to need, demand, and 
program cost, though these criteria are used less frequently 
in those institutions where only internal or self-reviews are 
conducted. 

While overall evaluative criteria are quite similar across 
postsecondary institutions, the number, specificity, and 
relative weight assigned to the criteria are markedly 
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TABLE 3 
CRITERIA FOR EVALUATION IN 
ACADEMIC PROGRAM REVIEWS 



Quality 

• Quality of faculty 

• Quality of students 

• Quality of curriculum 

• Quality of support services (library, laboratories and 
equipment, physica! plant, computer facilities) 

• Financr- resources 

• Quality of program administration 

Need 

• Centrality to mission and other campus programs 

• Value to society 

Demand 

• Present and projected student demand 

• Demand for graduates 

Cost 

• Cost effectiveness 

• Nonpecuniary costs and benefits 



diverse. (As the next section discusses, the methods and 
techniques used to measure various criteria are also 
diverse.) Several institutional examples illustrate this 
diversity. 

As part of a program review process mandated by the 
Kansas Board of Regents, the University of Kansas 
reviews each of its programs every four years. The major 
purposes of the reviews are to inform the board about insti- 
tutional offerings, to improve management, to identify 
strengths, and to describe steps that can be taken to cor- 
rect deficiencies. The criteria used to guide the reviews 
include enrollment trends, consistency of the program's 
objectives with institutional objectives, curricular strengths 
and weaknesses, qualifications and responsibilities of the 
faculty and staff, students' characteristics and qualifica- 
tions, special support requirements, and program need 
(Kansas Board of Regents 1984). 

At Ohio State University, quality is ;the major criterion 
for evaluation, but attention is given to other criteria, such 
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as the effective use of resources and the value of the pro- 
gram. The criterion of value refers to such factors as the 
market for program graduates, the contribution of the pro- 
gram to the university's instructional program, and the 
relationship of program goals to societal needs (Office of 
Acadeiwic Affairs/Graduate School 1978). 

At Southern Illinois University at Edwardsville, five 
major criteria are suggested for use in reviews of graduate 
programs: quality of instruction and learning, quality of 
faculty, centrality of the program, program value and/or 
uniqueness, and potential. While the same five criteria are 
employed across all program reviews, however, the rela- 
tive weight assigned to each criterion may vary by pro- 
gram. The relative weight (a numerical score) assigned to 
each criterion is determined through negotiation between 
administrators of each program under review and the 
appropriate review committee. Thus, depending on pro- 
gram objectives, the relative importance placed on the var- 
ious criteria may vary. For example, the rating criteria 
used to assess one program might emphasize quality of 
instruction and program centrality; for another program, 
greater emphasis might be placed on quality of the faculty 
and program potential (Russo, Brown, and Rothweiler 
1977, pp. 293-94). 

Methodological Approaches 

Probably the most controversial aspect of designing aca- 
demic program reviews concerns the methods used to eval- 
uate programs. At least partly as a consequence of this 
controversy, institutions frequently employ a combination 
of methods and techniques. Nevertheless, the majority of 
colleges and universities emphasize either Quantitative or 
qualitative methods that, in turn, are associated with quan- 
titative and qualitative approaches to evaluation. 

Table 4 compares the quantitative and the qualitative 
approaches to program review. Neither the table nor the 
following discussion should be taken to suggest that these 
two approaches are mutually exclusive, much less 
mutually antagonistic. Rather, quantitative and qualitative 
approaches can complement each other, and many institu- 
tions combine the two approaches in their reviews. More- 
over, while the table and the discussion identify various 
characteristics to distinguish the two approaches, these 
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TABLE 4 

COMPARISON OF QUALITATIVE AND 
QUANTITATIVE APPROACHES 
TO PROGRAM REVIEW 



Predominant 
Use 



Quantitative 

Surnmative 
Evaluation (decision 
making and 
accountability) 



Qualitative 

Formative Evaluation 
(program 
improvement) 



Evaluation 
Models 



Goal-based Model 
Decision-making 
Model 



Responsive Model 
Connoisseurship Model 



Orientation and 
Emphases 



Hypothetic-Deductive 

Verification 

Objective 

Scientific 

Quantification 

Fixed Design 
Statistical Analysis 



Inductive 
Discovery 
Subjective 
Naturalistic 
Description and 
Interpietation 
Emergent Design 
Holistic Analysis 



Methodological 
Approach 



Methodological 
Techniques 



Scientific Method 



Questionnaires 
Tests 
Records 
Unobtrusive 

Measures (objective 

indicators) 



Naturalistic Inquiry 
Ethnography 
Participant Observation 
Grounded Theory 
Ethnomethodology 

Personal Observation 

Interviews 

Field Study 

Documents 

Records 

Unobtrusive Measures 



Source: Adapted from Conrad 1982; Glaser and Strauss 1967; Guba and 
Lincoln 1981; House 1978; Mims 1978; Patton 1980, 1981, 1982; Rogers 
and Gamson 1982; Rossi and Freeman 1982; Weiss 1972. 
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characteristics do not fall within the exclusive domain of 
either approach. For example, while the goal-based model 
is commonly associated with the quantitative approach, 
many institutions use a goal-based approach to evaluation 
that relies on qualitative as well as quantitative methods. 
Therefore, the characteristics identified with each ap- 
proach should be viewed only as representing differences 
in orientation and emphasis. 

Quantitative evaluation has a long history in higher edu- 
cation and is the most widely used approach today. 
Accreditation, for example, has historically emphasized 
quantitative measurement of program quality; similarly, 
program reviews at the state and institutional levels have 
emphasized quantitative techniques. Briefly, the quantita- 
tive approach is based on the natural science paradigm of 
hypothetic-deductive methodology, in which the scientific 
method is adapted to program evaluation. In the quantita- 
tive approach, the evaluative criteria or varices are iden- 
tified and quantitative measures for those variables 
selected. The repertoire of quantitative measures includes 
questionnaires, tests, records, and objective indicators. 

As applied in academic program reviews, criteria and 
standards are identified in design of the evaluation. As dis- 
cussed previously, these criteria usually include such fac- 
tors as cost, demand, need, and quality. In turn, one or 
more measures or indicators are selected for each crite- 
rion. For example, three frequently used quantitative mea- 
sures of program demand include number of program grad- 
uates per year, number of majors in a program, and pro- 
gram credit-hour production. Because some criteria lend 
themselves better than others to quantification, institutions 
using a quantitative approach sometimes choose their crite- 
ria in concert with their selection of quantitative measures. 

An example of an institution that uses quantitative infor- 
mation extensively in evaluation is San Mateo Community 
College, which has developed a three-phase process for 
evaluating academic and occupational programs. The first 
phase is an efficiency study involving a review of program 
enrollments over two years and a proxy measure for the 
program's income and expenditures. This proxy is the ratio 
of faculty contact hours per year (expenditures) to weekly 
student contact hours (revenue). A unit that falls below a 
specified threshold on either measure becomes a candidate 
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for a Phase II review. The District Program Review Com- 
mittee coordinates the Phase II review and collects both 
quantitative and qualitative data: student flow, market 
demand, course retention rates, cost analyses, faculty sur- 
vey results, facility inventory, course scheduling prrctices, 
and program quality. After reviewing these data, the com- 
mittee recommends areas for improvement. "If problems 
are not remedied within one year, the program will enter a 
Phase III review, an administrative recommendation to 
retain, consolidate, or eliminate the program" (Smith 
1981, p. 2). 

At the University of North Dakota, the graduate school 
has established a departmental self-evaluation procedure 
that places primary evaluative responsibility for graduate 
programs within the departments and relies heavily on 
quantitative information. As part of each review, a two- 
page, quantitative summary of significant trends is devel- 
oped and a survey of program graduates conducted. This 
information is provided to each department and is used to 
respond to several aspects of a questionnaire that the grad- 
uate school has prepared. The graduate school also sur- 
veys all faculty members in a program and may, if deemed 
necessary, appoint an external review committee to help 
with the evaluation (University of North Dakota 1983). 

In marked contrast, the qualitative approach to evaluation 
draws from a variety of epistemological and methodological 
traditions. The qualitative approach plants its roots most sol 
idly in the traditions of ethnography and field study in anthro- 
pology (Pelto and Pelto 1978) but also draws from a variety 
of other perspectives, including phenomenology, symbolic 
interactionism (Denzin 1978), ethnomethodolpgy (Garfinkei 
1567), and ecological psychology (Parker 1968). In turn, a 
like diversity of qualitative methods exists: na* uralistic 
inquiry, ethnography, and grounded theory. 

While methodological differences clearly exist between 
the various qualitative evaluation strategies, all qualitative 
methods are holistic-inductive in that the evaluator seeks 
to generalize about the program ucier review within its 
natural setting. Unlike the quantitative approach, in which 
the design and evaluation criteria are determined in 
advance and guide data collection and analysis, the qualita- 
tive approach normally does not predetermine what vari- 
ables are worth measuring. Instead of gathering quaiitita- 
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tive data through predetermined scales or indicators, the 
qualitative researcher remains open to whatever emerges 
from the data. 

In contrast to [the quantitative approach that measures] 
the relationships among a few carefully selected and 
narrowly defined variables, the holistic approach to 
research design gathers data on any number of aspects 
of the setting under study in order to assemble a com- 
plete picture of the social dynamic of the particular situ- 
ation or program. This means that, at the time of data 
collection, each case, event, or setting under study is 
treated as a unique entity, with its own particular mean- 
ing and its own constellation of tdationships emerging 
from and related to the context within which it exists 
(Patton 1982, pp. 9-10). 

As the qualitative inquiry progresses and as patterns 
emerge from the data, the evaluator constantly moves back 
and forth between the discovery of underlying patterns and 
the verification 01 those patterns, thereby gradually weav- 
ing a tapestry intended to provide a holistic interpretation 
of the program. 

Especially under field conditions, qualitative inquiry 
replaces the "static snapshots" of the quantitative 
approach with a process-oriented approach to evaluation. 
Accordingly, qualitative researchers rely most heavily on 
personal observation and interviews but may also use rec- 
ords, correspondence, and other unobtrusive measures. 
Hence, in qualitative inquiry, qualitative data may: 

consist of detailed descriptions of situations, events, 
people, interactions, and observed behaviors; direct 
quotations from people about their experiences, atti- 
tudes, beliefs, and thoughts; and excerpts or entire pas- 
sages from documents, correspondence, records, and 
case histories (Patton 1982, p. 5). 

These data provide the raw material for program evalua- 
tion as the evaluator seeks to construct his nonquantitative 
portrayal of a program through the process of qualitative 
inquiry. 



No evidence suggests that any institutions use a purely 
qualitative approach to evaluation, but most use it in part. 
The use of external evaluators by the California State Uni- 
versity at Long Beach illustrates well the qualitative tech- 
nique (Office of the Associate Vice President n.d.). At that 
institution, evaluators perform a variety of functions, one 
of the most important being to synthesize a variety of 
information and provide a general assessment of a program 
that is based as much on experience and judgment as it is 
on quantitative indicators. Another frequently used qualita- 
tive technique is the internal review team. At the Univer- 
sity of Pittsburgh, an internal review team comprised of 
university faculty is a central feature of that institution's 
process for evaluating graduate programs. The qualitative 
aspect of the team's work relates to the meetings held with 
administrators, faculty, and students connected to a pro- 
gram and to the discussions held with the external review 
team. Such meetings are considered to be es^ntial if good 
judgments are to be made about such matters as a pro- 
gram's responsiveness to change, quality of treses, preem- 
inence of the faculty, and program strengths (University 
Council 1980) . It would be difficult to make judgments 
about these dimensions with quantitative measures alone. 
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MAJOR ISSUES IN PROGRAM REVIEW 



Six issues concern the individuals in colleges and universi- 
ties who are involved in academic program review: (1) 
accommodating multiple purposes; (2) selecting an evalua- 
tion model; (3) assessing quality; (4) using external review- 
ers; (5) increasing use of evaluations; and (6) assessing the 
impact of evaluations. (The selection of these six issues 
was based on two considerations— the significance of the 
issue and the extent to which a review of the literature 
might aid in illuminating its dimensions.) This chapter 
defines and clarifies these issues and presents alternative 
perspectives regarding their resolution. Those engaged in 
program review need to address these issues regularly. 

Accommodating Multiple Purposes 

Because a number of reasons usually exist for establishing 
a program review, institutions have developed a fairly 
lengthy list of purposes to guide their review efforts. The 
following list represents those adopted by many institu- 
tions: 

• to assess program quality, productivity, need, and 
demand 

• to improve the quality of academic offerings 

• to ensure wise use of resources 

• to determine the program's effectiveness and to con- 
sider possible modifications 

• to facilitate academic planning and budgeting 

• to satisfy state-level review requirements. 

The advantage of designing an evaluation system that 
incorporates all of these purposes is that it will appeal to 
several constituencies. Such support is often needed in the 
early stages of implementation, but it can become self- 
defeating. It is possible, however, to design an evaluation 
system in which information is collected and judgments 
made that respond to widely different expectations. 

Serious questions can be raised about two accommoda- 
tions of purposes. The first involves institutional efforts to 
combine program improvement and resource reallocation 
as major purposes of the review. The second concerns an 
institution's attempts to use a single review process to sat- 
isfy both its own review agenda and that of a state-level 
coordinating or governing board. 
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Combining program improvement with resource reallocation 
Although many institutions conducting program reviews 
emphasize either the improvement of program quality or 
resource reallocation (including program discontinuance), a 
growing number of institutions combine both emphases 
into a single process. At first glance, the two purposes 
would seem compatible. Each requires an assessment of 
current quality to identify where strengths exist and where 
improvement is warranted. Most people also would agree 
that quality assessment is an important first step in decid- 
ing how to reallocate resources. The problem, however, is 
that it is difficult, if not impossible, for those responsible 
for the review process to achieve both objectives at once. 
"Quality assessment should not be ignored in a retrench- 
ment process, but the two distinct motivations of improve- 
ment and reduction in resources will generally involve 
somewhat different processes and may produce quite dif- 
ferent results 1 ' (George 1982, p. 45). Several distinct diffi- 
culties in merging these two purposes merit consideration. 

A major problem in attempting to achieve both purposes in 
a single review system is that the underlying assumptions and 
ultimate objectives may not be easily reconciled, if at all. 
When an institution wishes to assess the quality of its pro- 
grams and to improve programs where weaknesses appear, 
the emphasis of the evaluation will be on how to assess cur- 
rent performance, on what progress has been made over 
time, and on what institutional strategies might facilitate fur- 
ther improvement. At the University of North Carolina- 
Asheville, for example, program improvement, not resource 
allocation, was the principal incentive behind recent program 
reviews. After studying a range of data, nine department 
chairs rated the institution's programs on the basis of nine 
criteria. Future resources of the institution were earmarked 
for programs with low rankings to promote program 
improvement (Cochran and Hengstler 1984). 

On the other hand, the approach to evaluation is likely to 
depart from this strategy if an institution concludes it must 
reduce the number of program offerings or reallocate 
resources away from low-priority or low-demand pro- 
grams. In this case, evaluators must judge relative worth 
and value and often must act quickly in the face of immedi- 
ate budget pressures. In this situation, quality is only one 
of several factors that gets considered. 
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In distinguishing between the two types of evaluations, it 
is useful to consider the "industrial" and "biological" 
models of evaluation (Pace 1972). The industrial model 
focuses on quantitative measures used to judge efficiency 
and productivity, and the biological model searches for 
ways to enrich experiences and to assess broader and more 
enduring program effects on students and society (Pace 
1972). It is not stretching this analysis too far to suggest 
that most resource reallocation processes are frequently 
industrial in orientation, while program improvement pro- 
cesses are more biological. Others have questioned the 
compatibility of the two purposes, pointing out the great 
difference between a unit thinking it is being evaluated to 
identify areas needing additional strength and units think- 
ing such an assessment serves to identify areas where qual- 
ity is low and support should be diminished or even elimi- 
nated (Arns and Poland 1980). 

The discussion of the review activities of the Select 
Committee on Academic Program Priorities at SUNY- 
Albany supports this view: "The Select Committee . . . 
felt a certain frustration that the budget recommendations 
had created a climate that led to the implementation of its 
negative recommendations but not its positive ones" 
(Volkwein 1984, p. 393). 

Yet another problem in merging these two purposes is a 
temporal one: The careful assessment of quality and the 
development of recommendations for improvement require 
considerable time. Most institutions are unable to conduct 
more than six or eight program reviews each year. Such a 
protracted schedule does not mesh well with resource real- 
location reviews, which usually transpire within relatively 
short periods in response to anticipated budget problems. 
The latter reviews use extant information on quality but 
usually lack sufficient time to conduct detailed analysis of 
the quality of every program offered. Hence, a timing fac- 
tor militates against efforts to combine the two types of 
reviews. 
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Accommodating institutional and state-level purposes 
A second accommodation of purposes concerns how insti- 
tutional review processes relate to the expectations for 
review of state higher education agencies. State agencies' 
authority and activity in program review have increased 
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dramatically in the last few years, and the proper role of 
state boards in institutional review processes continues to 
be debated. 

A key issue in this debate is whether institutional and 
state-level interests converge or are sufficiently comple- 
mentary so that a single review process can achieve the 
aims of both. Some believe that state-level interests can 
indeed mesh with those of institutions (Hines 1980). In 
Hines's view, the predominant institutional interest lies in 
assessing merit, while state agencies are most interested in 
statewide needs or plans. Although the primary objective 
of review differs for the two groups, the secondary objec- 
tives overlap considerably; that is, states have more than 
passing interest in merit, and institutions frequently want 
to know how programs could be more responsive to state- 
wide needs. 

An analysis of the relationship between program review 
for institutions and for state-level agencies identified eight 
purposes for undertaking program reviews and discussed 
how review responsibility varies by purpose (Wallhaus 
1982) (see table 5, pp. 44-5). In this view, responsibility for 
review is vested in the state agency when the review serves 
to develop a statewide plan or overall programmatic priori- 
ties. Conversely, reviews focusing on curricular and per- 
sonnel matters are the province of institutions. Between 
these two extremes exist a number of purposes where the 
assignment of responsibility is not so straight-forward. It is 
these purposes that institutions and state agencies fre- 
quently try to accomplish cooperatively through a single 
review process. 

These cooperative efforts have been labeled "shared 
reviews" (Floyd 1983). Responsibilities in shared reviews 
are defined variously. Frequently, the institution conducts 
the review and endeavors to attend to matters of interest to 
both the institution and the state. On the surface, this plan 
would seem to be reasonable and efficient. The issue, 
again, is whether a single review process can have more 
than one driving purpose. 

In the last few years, a number of states have adopted 
shared responsibilities for reviews. Illinois, Idaho, New 
Mexico, California, Oregon, and Ohio have adopted review 
processes in which the state agency, rather than conduct- 
ing reviews itself, simply ensures that each institution is 
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doing so (Barak 1982). In a comparable arrangement within 
the University of Wisconsin system, each institution is 
required to have a review process and must report its find- 
ings to the system office (Craven 1980b). Only in unusual 
circumstances, such as when enrollments arc low or when 
unnecessary duplication seems to exist, does the system 
office conduct a review of its own. 

In shared reviews, then, institutions attempt to accom- 
modate institutional and state-level purposes by developing 
an evaluation process that satisfies both institutional and 
state board requirements. In theory, the institution designs 
an evaluation system to meet its own needs and, by making 
minor adjustments in, say, data collection or reporting 
lines, is able to satisfy state-level needs as well. Moreover, 
this strategy minimizes duplication of review activities. 

Despite these positive features, the shared review 
approach has drawn criticism. While combining reviews 
appears to be a solid idea, theoretical and practical reasons 
militate against the success of such an approach (Barak 
1982, p. 84). For example, whereas institutions frequently 
establish the need for a program on the basis of students' 
or faculty members' perspectives, state agencies usually 
look at program need from a societal or manpower per- 
spective. Another example of potential conflict centers on 
efficiency. Institutions tend to assess efficiency by compar- 
ing similar departments on campus or by collecting data 
from peer departments. State agencies tend to look at effi- 
ciency from a statewide perspective, for example, how 
costs in a discipline vary among institutions in the state. 
Thus, separate reviews by institutions and state agencies 
may be the best solution (Barak 1982, pp. 84-88). 

Under certain conditions, combined reviews may not be 
productive (Wilson 1984). If the system office or state-level 
board prescribes the review process in too much detail, 
local initiatives aimed at establishing an effective system 
may be stifled. Where the state agency insists upon close 
adherence to a prescribed format, it may be more useful 
for separate reviews to be conducted, one by the institu- 
tion for its own use, another by the state agency. 

Selecting an Evaluation Model 

The key issue for evaluators is which model— goal-based, 
responsive, decision-making, and connoisseurship— should 
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TABLES 
PURPOSES AND OBJECTIVES OF 



• Determination of statewide educational policies, long-range plans, and 
programmatic priorities (that is, support development of statewide master 
plans) 

• Elimination of unnecessary program duplication or, conversely, 
identification of needs for new programs 

• Determination of educational and economic priorities in terms of: 

1. consistency with role and mission 

2. need for improvement or expansion and additional resources necessary 

to accomplish (link to budget decisions) 

3. decisions to decrease or terminate (link to resource reallocation 

decisions) 

• Determination of relationship to established standards of quality, or 
preparation for entry into professions, and so on (link to accreditation, 
continuation of operating authority, or licensing authority) 

• Improvement of communications with constituents; assurance that 
information provided to students, prospective students, parents, alumni, 
governmental agencies, and others is consistent with actual practice 

• Determination of quality controls and policies (for example, admission 
—'jj- graduation requii ements) 

ERIC 



REVIEW 



Tends to be more 
closely tied to 
state-level 
responsibilities 

X 



Tends to be more 
closely tied to 
institutional 
responsibilities 



X 



X 
X 
X 
X 



X 



X 



• Determination of curricular modifications, advisement procedures, X 
institutional plans, and priorities relative to instructional, research, and 

service objectives 

• Personnel and organizational decisions— faculty promotion and tenure, X 
academic leadership, organizational structures, and philosophies 

Source: Wallhaus 1982, p. 77. 
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guide program reviews in higher education, and the deci- 
sion is an important one: 



Where an administrator has a choice with regard to the 
type of evaluation to be conducted . . . that individual 
should be fully aware of the fundamentally different 
assumptions and outcomes that obtain when a particular 
type of strategy is selected (Gardner 1977, p. 572). 




Goal-based model 

The goal-based model focuses principally on assessing the 
extent to which a program's formal objectives are being 
achieved. As part of this effort, considerable attention is 
given to specifying program objectives, establishing stan- 
dards of performance, identifying data to assess perfor- 
mance, and evaluating whether objectives have been 
achieved. 

A goal-based approach to evaluation offers a number of 
positive features. First, the importance placed on objec- 
tives focuses attention on what those responsible for a pro- 
gram hope to accomplish. These goal statements become 
more than general statements of intent; they are specified 
as precisely as possible because of their significance to the 
subsequent assessment of performance. Second, a goal- 
based system can be used to make periodic checks of prog- 
ress (formative evaluation) as well as to make consummate 
judgments of program worth (summative evaluation). 
Third, the approach encourages systematic attention not 
only to whether program goals have been reached but also 
to those features contributing to success or failure. For 
example, if the desired goal is to increase student retention 
and if certain actions are taken to achieve this goal, the 
goal-based design would require both an assessment of 
whether retention had improved and an understanding of 
the effect on retention of the actions themselves. 

To be sure, the goal-based model has some limitations. 
The specification of goals can become an obsession result- 
ing in lengthy lists covering every conceivable desire, sig- 
nificant as well as trivial, for a program. "Some people 
believe that when every objective is related to every other, 
the program is properly arranged" (House 1982, p. 10). But 
"a m^jor criticism of evaluation as congruence between 
performance and objectives is that a focusing on measur- 
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able products rather than processes occurs. This may per- 
mit the overlooking of important side effects" (Feasley 
1980, p. 9). In essence, this criticism berates the propensity 
of ^valuators to focus on whether goals have been accom- 
plished while ignoring other, unintended contributions that 
goal statements have not captured. 

On a related matter, the goal-based system has been crit- 
icized because of the inflexible way in which a priori goals 
drive the process (Guba and Lincoln 1981). Evaluators 
have a tendency to accept the goal statements and to pur- 
sue data relating to those goals in a very determined way. 
If, in mid-evaluation, it becomes apparent that some of the 
goal statements no longer apply or that those responsible 
for the program should change directions, the goal-based 
evaluator might not perceive the necessary changes or, 
more significantly, might not be inclined to suggest that 
plans be changed, even if the need is apparent. Thus, the 
goal-based model sometimes engenders a single-minded 
pursuit of information relating to goal statements while 
ignoring everything else. 

Perhaps the major defect of the goal-based approach lies 
in its assumption that ways can be found to measure per- 
formance in relation to all goals. This observation is espe- 
cially significant for those institutions of higher education 
in which goals, such as those of program quality or central- 
ity, frequently are elusive. Attempts to "force" a goal- 
based model in a particular setting may result in redefining 
the goals in ways that can be measuied (for example, num- 
ber of publications equals research quality), thus trivializ- 
ing what is being done. 

Use of the goal-based model, therefore, offers the two 
advantages of a systematic attention to how a program has 
performed in relation to its intent and of a concern for the 
factors contributing to its success or failure. The model's 
chief limitations are the propensity to reduce everything to 
a goal statement, the insensitivity to outcomes that are 
unrelated to goal statements, and the assumption that valid 
measures can be found foi all goals. 

Responsive model 

The driving objective of the responsive model is to collect 
information to illuminate the concerns and issues of those 
who have a stake in an evaluation. Program goals are not 
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central. In essence, a responsive evaluation investigates 
what various constituents believe a program is accomplish- 
ing and their concerns about the program. 

The strength of this model is that it can help those 
responsible for a program to understand both its actual 
achievements and where action is needed to reconcile 
results with plans. For this reason, a responsive approach 
can be especially helpful during the early and middle stages 
of program implementation. 

One major criticism of the responsive model is that it is 
"unscientific" and lacks an emphasis on formal measure- 
ment (Gardner 1977). Critics suggest that the role of the 
evaiuator is not to observe a program from afar and to 
make judgments based on the analysis of "objective" data 
but to become immersed in the program to the point of 
rendering an accurate description and interpretation of its 
accomplishments. Such immersion, however, may well 
sacrifice objectivity, and it certainly increases the time 
commitment to a review. 

One particular type of responsive evaluation — the case 
study — has some definite shortcomings: the fact that the 
evaiuator assumes enormous responsibility in trying to por- 
tray a program accurately; the difficulty for the evaiuator 
to protect against bias; the requirement for a large number 
of subjective judgments (House 1982). 

Decision-making model 

The main purpose of the decision-making model is to con- 
duct evaluations responsive to the informational needs of 
decision makers. The strength of this model derives iVom 
the explicit connection between evaluation and decision 
making, a link that focuses the evaluation and increases the 
likelihood that results will be used. 

"The principal criticism of the decision-oriented 
approach is that the evaiuator accepts the decision context 
and values/criteria that have been defined by the decision 
makers" (Feasley 1980, p. 10). This criticism implies that 
the evaiuator is aligned with the decision makers and may 
find it difficult to remain objective. The evaiuator collects 
data according to the questions defined by decision makers 
and accepts the values implicit in their questions. If those 
responsible for the program do not share such values or 
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consider the questions unimportant, the evaluation will not 
be credible. 

Another problem is that this model assumes rational 
decision making (Gardner 1977). The task of the evaluator 
is to identify the questions of interest, collect pertinent 
information, present findings, and wait for the results to be 
used. The evaluation therefore becomes a critical ingredi- 
ent in decision making. This approach is likely to overem- 
phasize the importance of evaluative information and to 
fail to recognize that evaluations provide only one source 
of data for decision makers. Further, to assume that all 
decision alternatives can be accurately anticipated and that 
sufficient data can be collected in relation to these alterna- 
tives is to place unrealistic expectations on an evaluation. 

Decision makers are frequently biased in their acquisi- 
tion and processing of information (O'Reilly 1981, pp. 56- 
57), and this bias occurs in the search for information, in 
the preference for information that is easy to secure and 
supports preconceived ideas, in the transmission of infor- 
mation that distorts reality if it optimizes certain outcomes, 
in the selective use of available information, and in the 
preference for vivid examples, even if they arc misleading. 

A final criticism of this model relates to an evaluator' s 
ability to identify decision makers. It is not easy to identify 
decision makers in many complex organizations (Guba and 
Lincoln 1981), Decisions are frequently made at several 
organizational levels, by various individuals. Most actions 
involve more than one decision maker and a number of key 
decision points. It is almost impossible to identify all cf 
these individuals and to collect all of the data necessary to 
inform them. 
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Connoisseurship model 

The central tenet of the connoisseurship model is that an 
expert (a connoisseur) can use his experience and expertise 
to judge a program. In essence, the human being is the 
measurement instrument. 

The use of outside reviewers in higher education is a 
good example of the connoisseurship model. The strength 
of this model is that those who are most knowledgeable 
about a subject are asked to make the assessment. The 
connoisseurship model has high credibility because those 
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within a discipline or profession are judged by peers who 
have a sound basis for understanding what is — or is not — 
being accomplished. 

One problem is that the connoisseurship model fre- 
quently lacks evaluative guidelines, so that a premium is 
placed on the evaluator's judgment; it is hard to know 
whether the evaluator's perceptions are accurate (Guba 
and Lincoln 1981). Many institutions attempt to sidestep 
this problem by inviting more than one expert to partici- 
pate in a review, a strategy introducing valuable "triangu- 
lation." At the same time, however, this strategy can yield 
as many different assessments as there are evaluators. 

Another problem with this approach lies in the difficulty 
of generalizing across programs (Feasley 1980, p. 8). In rat- 
ing different programs, no two experts will have the same 
value structure or will weigh criteria equally. One evalua- 
tor may rate a program weak because of difficulties in its 
graduate instructional program; another may overlook the 
graduate program entirely if the record of faculty research 
is strong. 

The connoisseurship model is popular in higher educa- 
tion because most faculty members believe that only those 
within a discipline can adequately evaluate accomplish- 
ment. Certainly, a disciplinary background can greatly 
enhance an evaluation. At the same time, certain problems 
are inherent — the ability to generalize procedures across 
programs, the subjectivity of perceptions, and the empha- 
sis placed on the person chosen to conduct the evaluation. 

Assessing Quality 

For those engaged in program review, the assessment of 
quality has generated more confusion and debate than any 
other issue. Pressure always has existed to define "qual- 
ity" and to determine which types of information should be 
collected, but more recently, interest has burgeoned 
because of the emphasis on program review for realloca- 
tion and retrenchment. The problem is that no one has yet 
found a way to measure quality directly. The issue for 
evaluators is how to define this concept and how to deter- 
mine what types of information (indicators) should be used 
to guide data collection. 

The literature (cf. Astin 1980; Conrad and Blackburn In 
press b) and institutional documents identify four perspec- 
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tives on how to define quality: a reputational view, a 
resources view, an outcomes view, and a value-added 
view. The particular view held affects the kind of informa- 
tion used to assess quality. The issue is which of these 
views of quality is most accurate and helpful (see table 6). 

TABLE 6 

VIEWS OF QUALITY AND 
REPRESENTATIVE INDICATORS 

Reputational View Outcomes View 

• Peer judgments of the # Faculty scholarly productivity 
quality of program, # Facu , ty awards ^ honors 
students, faculty, or # Facu!ty research supnon 
resources # Faculty teaching performance 

e Student achievement following 
Resources View graduation 

• Student selectivity • Student placement 

• Student demand • Student achievement 

• Faculty prestige © Alumni satisfaction 

• Faculty training 

• Faculty teaching loads Value-added View 

• Budget affluence # Change in stude nts' cognitive 

• Library holdings AMes 

• Equipment adequacy # Student personal development 

• Size of endowment # Student career dev elopment 

• Social benefits 

Reputational view 

This view of quality is derived from the connoisseurship 
model of evaluation and assumes that experts in the field 
make the best judgments on the criterion. In essence, the 
reputational view reflects a belief that the optimum way to 
assess quality is to seek a consensus of informed opinion. 
The typical indicator is some type of reputational survey. 
The past two decades have seen a number of surveys of 
this lype (Cartter 1966; Jones, Lindzey, and Coggeshall 
1982; Roose and Anderson 1970). 

The main strength of this view lies in the fact that the 
raters are those who supposedly know best what quality is. 
It also has an intuitive appeal to ratings, reflecting what 
most people believe is true (Webster 1981). 

Reputational rankings are criticized, however, because, 
while the raters may have insight into the scholarly produc- 
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tivity and reputation of a department, they are not likely to 
know much about the instructional program. Surely a pro- 
gram assessment must include more than research and 
scholarship (Conrad and Blackburn 1985). The lack of 
national visibility for many programs suggests that even 
reputational rankings based on faculty members' scholarly 
productivity are not likely to be meaningful below the top 
15 or 20 programs in the country (Webster 1981), Other 
problems are apparent — "reputational lag" (the ranking of 
programs based on their quality several years ago) and 
"halo effects" (ranking a program high because the institu- 
tion is held in high regard). These and other limitations are 
discussed extensively in the literature (Conrad and Black- 
burn 1985; Dolan 1976; Lawrence and Green 1980; Web- 
ster 1981). 

Despite these limitations, such ratings have received 
support: 

In our view controversy over reputational studies should 
not deter researchers from conducting such studies in 
the future. If reputational studies are designed to 
respond to the criticisms . . . we are persuaded that they 
can make an important contribution to the evaluation of 
quality in higher education (Conrad and Blackburn 1985, 
p. 23). 




Resources view 

This particular view of quality emphasizes the human, 
financial, and physical resources that go into a program. 
According to this view, high quality exists where these 
resources — bright students, excellent faculty, adequate 
budgets, strong research support, strong libraries, and ade- 
quate facilities — are plentiful. The extent to which these 
resources are available to a particular program has been 
measured in various ways — for example, student test 
scores, proportion of the faculty with a doctorate, grant 
support, and number of volumes in the library. The advan- 
tages of using such measures of resources are that relevant 
data are readily available at most institutions, that the mea- 
sures reflect what exists today, not what the situation was 
a decade ago, and that comparisons can be made across all 
colleges and universities, not just a few highly ranked insti- 
tutions (Webster 1981). 
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Notwithstanding these benefits, the resources view suf- 
fers some serious limitations. Little evidence supports the 
view that more resources equate with increased student 
learning (Astin 1980). Further, and more important, the 
resources approach places a false ceiling on the amount of 
quality that can exist in higher education by asserting that 
"such resources as bright students and prestigious faculty 
are finite" (Astin 1980, p. 4). 

Outcomes view 

Another way to define and assess quality is to emphasize 
results — what the investment of resources produces. Here, 
attention is focused on the quality of the product. Typical 
indicators associated with this view are faculty productiv- 
ity , students' accomplishments following graduation, 
employers' satisfaction with program graduates, and insti- 
tutional contributions to the solution of local, state, or 
national problems. Specific outcome measures include the 
number of faculty publications in scholarly journals, the 
number of graduates admitted to leading graduate or 
professional schools, employer surveys, percentage of 
graduates finding employment soon after graduation, and 
lifetime earnings of graduates. 

Collecting information on outcomes boasts a number of 
advantages. Chief among them is the emphasis on what is 
happening to those who are or have been part of a pro- 
gram; the focus of attention shifts from the resources 
invested to the results. Like the resource measures, many 
of the outcomes measures hold relevance for all institu- 
tions; all institutions, for example, are interested in the 
accomplishments of their alumni (Webster 1981). 

Perhaps the most significant problem with the outcomes 
view is the difficulty of delineating the special institutional 
contribution to results. "Most output measures depend 
more on the quality of students admitted to the institution 
than on the functioning of the institution or the quality of 
its program" (Astin 1980, p. 3). Another disadvantage is 
that outcomes measures frequently limit themselves to the 
past. The period between graduation and inclusion in 
Who's Who in America obstructs the drawing of precise 
conclusions about the current quality of a program (Web- 
ster 1981). 
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Value-added view 

This view of quality focuses attention on program impact. 
"The basic argument underlying the value-added approach 
is that true quality resides in the institution's ability to 
affect its students favorably, to make a positive difference 
in their intellectual and personal development" (Astin 
1980, pp. 3-4). Consonant with this view, evaluation 
should attempt to identify what a program contributes to 
students* learning. One typical indicator is what students 
learn while enrolled, which is sometimes measured by 
administering an achievement test at the time of enrollment 
and at graduation. 

The chief advantage of this view is that one takes into 
account the quality of students at entry. This approach is 
especially attractive to institutions seeking to respond to 
"the twin doctrines of entitlement and equal education 
opportunity" (Lawrence and Green 1980, p. 54). Thus, 
institutions are judged by much they help students, by 
how much they "add" to sludents' knowledge and per- 
sonal development. 

Like the other views of quality, the value-added 
approach has limitations. First, it is expensive, both in 
time and money. Investigating a program's contribution 
requires extensive recordkeeping for a large number of stu- 
dents. Another problem is the difficulty of reaching con- 
sensus on what students should learn and on measuring 
such quantities, even if they are defined (Lawrence and 
Green 1980, p. 40). For example, significant measurement 
problems are associated with assessing how much a stu- 
dent has improved in critical ft * ''king skills. Finally, it is 
no easy matter to determine w^at one program's contribu- 
tion is to a student's learning or development. The effects 
of other variables, such as maturation, travel experiences, 
and summer employment, are difficult to control. 

Using External Reviewers 

This review of current evaluation processes indicates that 
most institutions have incorporated the judgments of exter- 
nal reviewers into their program reviews. Most often, these 
reviewers are faculty members within the same discipline 
but at another institution or within the institution but out- 
side the program under review. The issue faced by those 
designing an evaluation system is to decide which of these 
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two types of reviewers to use. Knowledge of the possible 
strengths and limitations of each should prove helpful in 
making the choice. 

Reviewers from other institutions 

The use of peers from other institutions to help in institu- 
tional reviews is rapidly becoming the norm rather than the 
exception. The program review process at California State 
University at Long Beach illustrates the use of outside 
reviewers. That institution's review process consists of 
four phases: (1) a self-study prepared by those in the pro- 
gram under review; (2) a review by a subcommittee of fac- 
ulty from other programs on campus; (3) ar* external 
review by disciplinary experts; and (4) a "response report" 
prepared by those in the program (Office of the Associate 
Vice President n.d.). The external review serves to provide 
a comparative perspective, which is balanced with the pro- 
gram's own view and that of colleagues on campus. 

Like other approaches to assessing program quality, the 
use of external peers has its strengths and weaknesses. 
Characteristic of the problems are the following observa- 
tions on experiences with outside reviewers at the Univer- 
sity of Nebraska-Lincoln: The selection of a review team 
was frequently controversial; the review teams suffered 
because of lack of knowledge about the local context; too 
little time was available for the reviews; the review teams 
tended to focus on insignificant issues; the review teams 
often were asked to address problems they could not 
resolve; the review teams were provided with more infor- 
mation than they could comprehend; and the review teams 
tended to solve all problems by recommending additional 
resources (Seagren and Bean 1981, pp. 20-24). 

In a more positive vein, the use of external peers pro- 
vides a perspective that is frequently helpful. In most pro- 
gram reviews, it is considered crucial to have some kind of 
disciplinary perspective on the quality of what is being 
done and to seek advice on future directions. In addition, 
reports from externrJ peers are usually perceived as objec- 
tive and therefore can stimulate change that might not 
otherwise be possible. 

Reviewers from the same institution 

A number of institutions choose to use on-campus (but out- 
side the discipline) colleagues to help evaluate programs. 
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At California State University at Long Beach, for example, 
faculty from within the institution conduct an internal 
review of a program to provide an assessment based on 
institutional (as opposed to disciplinary) standards of per- 
formance and quality (Office of the Associate Vice Presi- 
dent n.d., Appendix B, p. 14). This strategy offers the 
advantages of familiarity with the local context and norms 
and a stake in the results. The recommendations will affect 
not only those evaluated but also the evaluators — they 
must live with what they recommend. On the other hand, 
such reviewers may frequently be unfamiliar with the disci- 
pline under study or, conversely, tend to allow previous 
familiarity with a program or its personnel to bias results. 

Despite these criticisms, many believe that reviewers 
outside a particular discipline can recGgnize quality as long 
as enough information is available and enough opportuni- 
ties exist to interact with program personnel. An interest- 
ing tes: of this idea examined results of faculty ratings of 
students' oral examinations (DiBiaso et al. 1981). A gradu- 
ate school representative from outside the student's disci- 
pline was appointed to each of the review committees. A 
comparison of the ratings of the internal and external 
reviewers revealed 

no evidence of a significant difference between how 
graduate school representatives rate examinations con- 
ducted inside their own colleges compared to their rat- 
ings of examinations outside their own colleges. These 
results suggest that members have some common per- 
ceptions about the quality of doctoral examinations , 
regardless of discipline (DiBiaso et al. 1981, p. 10). 

Thus, the issue is not whether to use reviewers in higher 
education, but whether to use on-campus colleagues or dis- 
ciplinary peers. 

Increasing Use of Evaluations 

One of the most perplexing issues facing evaluators is how to 
increase the likelihood that others will employ <he results of 
their efforts. Considerable time and attention is being given 
to evaluation these days, yet a frequent criticism is that the 
results of such efforts really have no effect on decisions. The 
perception is that evaluations are undertaken not because the 
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results are expected to be used but because someone simply 
feels they "ought to be done." This criticism is so prevalent 
that it must be taken seriously. 

To the uninitiated, it would seem that the issue of use 
should not even arise. Is not the basis of an evaluation the 
need for information to make a decision or to become more 
knowledgeable about a program or activity? If so, then 
every evaluation should be used. Nevertheless, utilization 
is a problem. 

The results of program evaluations are not used for four 
general reasons: (1) organizational inertia; (2) the state of 
evaluation practice, for example, the inability to define 
valid measures of important criteria; (3) the uncertainty 
about the need for some evaluations; and (4) the multiple 
sources of information competing for the attention of deci- 
sion makers (Anderson and Ball 1978). 

The conflicting information needs of people at different 
levels of an organization make it difficult to conduct a use- 
ful evaluation (Patton 1985, p. 13). Highly detailed discus- 
sion of a specific case is seldom of much use at higher 
organizational levels; aggregate comparisons are of little 
use at the unit level. One important reason for lack of use 
of an evaluation is the inadequate personal involvement 
and commitment of key people. The personal factor is 
more crucial than structural, organizational, or method- 
ological variables (Patton 1981, pp. 15-16). 

Use of evaluations is also hindered because institutions 
compartmentalize the function of evaluation. Typically, the 
responsibility for program review is assigned to a staff 
office or to a specific individual. The delegation of respon- 
sibility for program review by the executive officers of an 
institution relieves them of the responsibility for such 
activities and places distance between those conducting the 
reviews and those in a position to use the results. 

Utilization is also impeded because decisions frequently 
involve social, political, and financial considerations out- 
side the task of evaluation. It should not be too disturbing 
to eyakiators to knew that occasionally these other consid- 
erations will outweigh the findings of an evaluation report 
(Dressel 1976; O'Reilly 1981). 

Given that these problems exist, the issue of how to 
increase the likelihood that results will be used warrants 
special attention. This matter has engendered a number of 
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views. Anderson and Ball (1978), for instance, recommend 
encouraging communication between those evaluated, 
those evaluating, and those responsible for the process; 
varying the modes for disseminating results according to 
audiences; identifying users early and finding ways to make 
sure that their questions are being addressed; finding ways 
to include those responsible for the evaluation in its plan- 
ning; reporting results in a timely way; and maximizing 
such virtues as brevity, timeliness, and responsiveness. 
Others suggest that utilization should become an immedi- 
ate rather than a postreport concern, that reports relate to 
the concerns of decision makers, that credibility and rap- 
port be maintained, and that all participants in the evalua- 
tion communicate among themselves (Brown and Bras- 
kamp 1980). "Evaluation is undertaken in a social and 
political environment in which various groups have vested 
interests in the evaluation process. . . . If an evaluation is 
to be used by these groups in their deliberations, discus- 
sions, and policy making, the evaluation system must be 
designed to maximize communication among these audi- 
ences" (Braskamp 1982, p. 58). 

A critical element in utilization relates to the approach of 
the ovaluator (Alkin 1980). In particular, rapport estab- 
lished with program staff can enhance use of results. Use is 
not related to any particular evaluation model; the most 
important consideration is to adapt the strategy to the pro- 
gram and to the questions being asked. 

One of the most important ways to increase utilization is 
for decision makers and information users to be clearly 
identified (Patton 1978). Decision makers cannot be treated 
as "abstract audiences" (p. 284). Decision makers should 
not delegate responsibility for an evaluation but should 
assume an active role in its implementation: 

There has been considerable discussion in the literature 
and among evaluators about how to make managers, cli- 
nicians, board members, and others better consumers of 
evaluations. . . . This effort is misplaced. For evalua- 
tions to be useful and to be used, the managers have to 
accept responsibility for owning and defining the evalua- 
tion function (Clifford and Sherman 1983, p. 32). 

One way to increase use is to ensure that evaluators 
focus their efforts on three issues: (1) who the decision 
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makers will be; (2) what information is needed; and (3) 
when the information is needed (Feasley 1980, p. 43). Per- 
haps the most important issue is to identify the evaluative 
question (Patton 1978). 

A study of the characteristic* zt decision makers identi- 
fies six managerial characteristics relating to the issue of 
utilization: 

1. Decision makers work at an unrelenting pace; 

2. their daily routines are characterized by brevity, vari- 
ety, and fragmentation; 

3. they prefer active rather than passive use of time; 

4. they prefer verbal as opposed to written communica- 
tion; 

5. they serve as active communication links; 

6. they blend rights as well as duties so that personal 
objectives can be realized (Mintzberg 1973, chap. 3). 

This list suggests that evaluations are more likely to be 
used if they relate to decision makers' concerns, are com- 
municated clearly and concisely, and are presented both 
verbally and in written form. 

In wrestling with the question of utilization, institutions 
have adopted several strategies to try to ensure that results 
of evaluations will somehow link to other decision-making 
processes. Ohio State University has developed the con- 
cept of a loosely coupled system, which means that all 
"parties to a review," including the college dean, the uni- 
versity's chief academic officers, the graduate dean, and 
those in the program, are consulted throughout the review 
process. At the conclusion of the review, a "memorandum 
of understanding" is developed in which the parties agree 
on actions to be taken. These agreements are monitored 
and updated each year (Arns and Poland 1980). 

At the University of North Carolina at Asheville, six 
aspects of the program review process contribute to its 
usefulness: 

1. clarity of purpose 

2. involvement of decision makers in all stages of the 
process 

3. maximization of communication 
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4. understanding of the political nature of the environ- 
ment 

5. recognition of the subjectivity of evaluation 

6. competence of the institutional research staff and 
confidence in the data collected (Cochran and Heng- 
stler 1984, p. 184). 

At the State University of New York at Albany, evalua- 
tions are an integral part of a planning process. The evalua- 
tions consist of both annual monitoring of programs and 
five-year in-depth reviews. This arrangement is useful 
because: 

1. it capitalizes on an annual, synoptic view across all 
major university activities; 

2. if is a goal-driven activity; 

3. it is merged with resource allocation, thereby linking 
budgeting with evaluation; 

4. evaluation (both ongoing annual monitoring as well 
as selected in-depth reviews) provides feedback for 
planning and resource allocation; 

5. it more clearly integrates evaluation with existing 
decision-making structures and processes (Hartmark 
1982, p. 16). 

Thus, a review of the literature on utiliza.ion reveals 
consensus on the objective of utilization but little agree- 
ment on how it is best accomplished, and suggestions vary 
from encouraging decision makers to participate more 
actively in evaluation to accounting for the managerial 
characteristics of decision makers to conducting evalua- 
tions in a manner responsive to those characteristics. 



Assessing Impact 

If results are used, another issue emerges— the impact of 
those results. The basic concern is whether the conse- 
quences of implementing an evaluation are positive or neg- 
ative. First, however, one must distinguish between the 
outcomes and the effects of a review— a subtle but impor- 
tant distinction. Decisions to eliminate a program, to 
increase admissions requirements, to change department 
heads, or to establish consortia arc outcomes, not effects, 
of program reviews. As defined here, "effect' ~fers to the 
consequences of actions taken. Concern abou; ue effects 
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therefore requires attention to the long-term effects of deci- 
sions, for example, whether the program is stronger, more 
efficient, or higher quality. Some believe the effects of pro- 
gram review are salutory; others are less optimistic. The 
question is which view is more correct. 

How does one make such assessments? Efforts should 
focus on the question, "Does the system function better as 
a result of the evaluation effort?" (Cronbach 1977, p. 2). 
Explicit in this question is the principle that an evaluation 
must be beneficial. 

The following kinds of consequences should be noted: 

The ideal held forth in the literature is one of major 
impact on concrete decisions. The image that emerged 
in our interviews is that there are few major, direction- 
changing, decisions in most programming. . . (Patton 
1978, p. 32). 

Most conceptions are too narrow (Alkin 1980). Conse- 
quences cannot be examined solely on the basis of immedi- 
ate impact; longer-term implications must also be consid- 
ered. Evaluations have unintended results that go beyond 
the formally stated recommendations. Further, the evalua- 
tion process often generates benefits beyond those chroni- 
cled in a report. Thus, the assessment of impact must be 
done in a naturalistic way, as in conducting case studies 
and recording participant observations. Finally, one should 
not confuse lack of implementation with lack of impact. An 
evaluation report frequently provides valuable information 
even though no specific recommendations are implemented 
(Alkin 1980, pp. 21-22). 

The assessment of impact must not be limited to immedi- 
ate and direct influences; indirect, catalytic, and inclusive 
results also demand attention (Braskamp and Brown 1980, 
p. viii). This approach requires special skill in analyzing 
multiple causes of specific actions as well as a willingness 
to view utilization broadly. Thus, those assessing impact 
must heed results that may be latent as well as immediate, 
incremental as well as radical, subtle as well as obvious. 
This view of impact is consistent with the admonition that 
"most change in education is incremental rather than radi- 
cal, and advertising of this fact would improve the climate 
for evaluation" (Diessel 19/6, p. 5). 
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Just as the advice on how to assess impact is far from 
uniform, so too is the evidence on impact far from defini- 
tive. Most campuses have critics who believe that the costs 
of program review outweigh the benefits. Several criticisms 
have been cited frequently: 

• Time and effort are wasted because more data are col- 
lected than can be productively used. 

• Viewed as inherently threatening and negative, the 
review process creates unwarranted anxiety. 

• Leaders' credibility is diminished because the infor- 
mation requested is not used, or its use is not made 
visible enough. 

• Distrust is created because the uses of the information 
are not conceived and articulated clearly enough from 
the outset, confidentiality of the report is not clarified, 
or the various roles in the process are not adequately 
determined. 

• Inaccurate information causes unwarranted embar- 
rassment or pride. 

• Attention and time are diverted from the institution's 
teaching, research, and service functions. 

• Resentment arises because the process is not designed 
to be useful at the program level as well as at higher 
organization? 1 , or system levels. 

• The review leads to raised expectations for resources 
that are unavailable, which causes disappointment 
(Seeley 1981, p. 56). 

On the other hand, program review— 4 4 if implemented 
properly and combined with other retrenchment strate- 
gies—can be a major tool for effectively reducing expendi- 
tures while maintaining essential program quality" (Barak 
1981 , p. 219). A study of program reviews in research uni- 
versities found, not surprisingly, that the benefits were 
greatest at the program level ard least at the institutional 
level (Poulton 1978). Table 7 displays the nature of the 
effects at three organizational levels. 

The results of graduate program reviews at the Univer- 
sity of California indicated several conclusions: (1) the 
institutions conducting reviews did not save money and, in 
fact, lost money (if the cost of the review process itself is 
taken into account); (2) revievs did not uncover previously 
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TABLE 7 

TYPICAL EFFECTS OF PROGRAM REVIEWS 



Organizational 
Level 

Department/Program 



Relative 
Utility 

Greatest Utility 
(primarily from 
single reviews) 



Nature of Changes 



School/College 



Moderate 
Utility 



University 
Administration 



Least 

Utility (requires 
accumulation of 
reviews) 



Increased introspection 
Revised objectives for 
teaching and research 
Better organized 
qualitative and 
quantitative information 
Clarified unit/program 
goals, strengths, and 
deficiencies 

Improved unit procedures 
Improved contact among 
unit members 
Improved rationale for 
resources 

Potentially increased 
frustrations 



• Improved information on 
unit trends and priorities, 
strengths, and weaknesses 

• Better indications of unit 
qual : ty and responsiveness 

• Adjusted college policies 
and procedures 

• Adjusted resource 
decisions (occasional) 

• Adjusted organizational 
structures (occasional) 

• Revised institutional 
policies and procedures 

• Major organizational 
changes (rare) 

• Major budgetary 
commitments or cuts (rare) 



Source: Adapted from Poulton 1978. 
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unknown information; (3) reviews did stimulate change in 
some situations; (4) the reviews did tend to clarify impres- 
sions and develop a fair portrayal of programs; and (5) 
many of the reviews' recommendations were implemented 
(Smith 1979, pp. 2-3). Overall, the study concluded that 
the institution benefited from the reviews. A similar assess- 
ment at the University of Iowa took place in 1977, when an 
ad hoc committee of the faculty senate was appointed to 
evaluate the program review process on campus (Barak 
1982). Through interviews with participants, the committee 
found that the reviews required a substantial commitment 
of time and effort but that many positive benefits accrued 
to the institution. The self-study process was found to ben- 
efit the units and to lead to improvements. The reviews 
also provided systematic information useful in keeping fac- 
ulty and administrators knowledgeable about programs. 

A survey of the program review authority and practices 
of 37 state-level higher education agencies, paying special 
attention to results in terms of resources, found: 

Despite the concern about resource savings, only one 
respondent supplied a dollar figure for resources saved. 
In fact, 95 percent of the 20 respondents who have dis- 
continued programs do not know the amount of 
resources saved or reallocated and only 35 percent of 
those same 20 respondents believe that resources have 
been saved, even though they could not supply a dollar 
figure (Skuball979,p. 231). 

On the basis of these results, one of three possible scenar- 
ios is taking place: (1) savings accrued at the institutional 
level, rather than at the state level; (2) it was impossible to 
attach a dollar amount to the savings; and (3) the review 
activity was purely cosmetic— programs being eliminated 
involved no resources (Skubal 1979). As far as the state 
boards were concerned, program discontinuance had not 
had a substantial impact on resources (p. 232). 

A study of the effects of program reviews conducted by 
state-level higher education agencies gave careful attention 
to the effects of the Louisiana Board of Regents' program 
reviews involving Louisiana State University-Baton 
Rouge (LSU) and Northeast Louisiana University (Mingle 
1978). At LSU, no cost savings were realized, but the 
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belief persisted that the reviews facilitated cooperation 
among programs, provided a basis forjudging programs* 
worth and for reallocating resources, stimulated personnel 
changes, and enhanced quality standards. By contrast, the 
reviews at Northeast were viewed as biased and as having 
fostered a "sense of declining prestige and fear for the 
future" (p. 64). 

Reviews conducted by the Florida Board of Regents used 
outside consultants to review selected programs in the nine 
universities of the system (Hill, Lutterbie, and Stafford 
1979). The consultants' task was to review a particular pro- 
gram at all of the institutions and to make recommendations 
on "program quality, duplication of programs, financial sup- 
port, and the need for any additional programs or a shifting 
of programs in the discipline under review" (p. 3). These 
reviews provided better documentation of the need for new 
programs, resulted in a small number of programs' being 
eliminated, and controlled program growth. Further, the 
reviews led to the establishment of several cooperative pro- 
grams among institutions within the system and between 
those institutions and private colleges and universities in the 
state. For example, an engineering consortium was estab- 
lished, and contracts between the state and several private 
colleges and universities were developed. Further, the 
reviews helped the board identify underfunded areas and pro- 
vided some systematic information that could serve as part of 
a recently initiated planning exercise (pp. 5-8). 

While several problems with the review process were 
noted in Florida, the overall assessment was quite positive. 
The consultants' reports were believed to have aided greatly 
in identifying the strengths and weaknesses of programs in 
the system and in stimulating plans to strengthen some pro- 
grams and to address important issues in others. "Some of 
the impact [of the reviews was] felt immediately by the uni- 
versities, but the lai^er impact [was] more subtle as the intri- 
cate process of change in a multicampus system [was] initi- 
ated, developed, and brought to conclusion" (p. 9). 

Although some evidence suggests that program reviews 
are helpful, the basis for this conclusion is weak, because 
only a few studies have examined effects systematically. 
Some evidence also suggests that such reviews do not 
achieve desired results. The stubborn fact is that not much 
is known about the effects of program review. 
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PROPOSALS FOR IMPROVING PROGRAM REVIEW 



This chapter offers some tentative proposals on ways to 
improve program review processes, both as a practical 
guide and as a research agenda. For practitioners, the pro- 
posals suggest solutions to some recurring problems, draw- 
ing heavily on available research and institutional experi- 
ences. For researchers, the proposals constitute a set of 
hypotheses that need to be tested to determine their valid- 
ity and utility. The proposals address a few important con- 
cerns rather than burying the significant among the pedes- 
trian in a more comprehensive list of suggestions. The pro- 
posals relate to three general aspects of program review: 
purposes, processes, and results. 

Purposes 

1. Campus-based reviews should have a single driving 
purpose that is widely accepted and understood. Multi- 
ple purposes can be achieved only to the extent that 
their underlying assumptions and objectives are 
mutually supportive. 

Many institutions have established review processes that 
encompass a broad range of purposes. These purposes 
often address fundamentally different objective?;, and the 
success of reviews may be compromised if they are pur- 
sued simultaneously. A good illustration of inherent con- 
flict can be found in reviews that seek both program 
improvement and resource reallocation. 

It is difficult to sustain a review process with conflicting 
purposes (Barak 1982; George 1982). The root of the prob- 
lem is often that the purposes of a review evolve in subtle 
ways. As the process is being developed, reviewers are 
inclined to broaden the purposes to achieve several objec- 
tives. Frequently, the broadening is well intentioned, aim- 
ing to increase the apparent efficiency of the process and to 
increase support for its implementation. Such accommoda- 
tions, however, may undermine the ultimate success of 
evaluations and should occur only after careful examina- 
tion of the compatibility of underlying assumptions and 
ultimate expectations. 

2. Institutions should develop a way to regularly and 
systematically assess program quality and to suggest 
ways for programs to improve. Although this effort may 
be linked to planning and budgeting processes > it should 
be a separate activity. 
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Writers genera^y agree that information on program 
quality and on strategies to improve them should be 
included in planning and budgeting processes (Craven 
1980b; Hartmark 1982; Volkwein 1984). The assessments 
required to produce such information, however, are too 
important (and too time consuming) to be treated as inci- 
dental activities. Annual budgeting and, to a certain extent, 
planning tend to overwhelm other considerations because < 
of the immediacy of the decisions being made. Information '* 
about program quality will be sought in varying degrees in J 
making such decisions, but it is unrealistic to assume that 
those responsible for planning and budgeting have the time 
and energy required for thorough assessments of quality. 
An institution's interests are best served if the assessment 
of quality and the development of budget plans are under- 
taken separately. 

Processes 

5. // is unlikely that institutions will be able to design an 
evaluation system that strictly conforms with any of the 
evaluation models suggested to date. The value of the 
models is not in the wholesale adoption of a particular 
one but in the selective use of features from several. 
This review of currently used program review models 
suggests that most are, to a large extent, based on objec- 
tives. Reviewers are logically drawn to an evaluation sys- 
tem that looks at congruence between original plans and 
subsequent performance. This appeal partly explains why 
the responsive (or goal-free) model has not been popular. It 
can be hard to attract support for an evaluation process in 
which program goals do not provide the centerpiece for the 
evaluation. What the responsive model can contribute to 
an evaluation system, however, is a sensitivity to unin- 
tended results and to the need to consider the perspectives 
of all constituencies. Thus, a feature of the decision- 
oriented model relevant for most institutions is the central- 
ity of questions that need to be answered as a result of the 
evaluation and of those who are in a position to use the 
results. Finally, the connoisseurship model adds the peer 
perspective, a critical element in making judgments about 
disciplinary matters. No single model is adequate: The use 
of features from several different models enriches evalua- 
tions and is more likely to yield useful results. 





4. Program review processes should be flexible enough 
to accommodate change once it appears necessary and 
should encourage interaction between those conducting 
the evaluation and those being evaluated. 

A review process must be flexible. It is impossible to 
design a review process that anticipates all the questions 
that need to be addressed and all the constituencies that 
need to be consulted. It is useful to have a standard evalua- 
tive procedure, but standard evaluations should probably 
not exist. Early attention will need to be given to how a 
standard procedure should be altered to make it relevant in 
a particular situation. These kinds of adjustments require 
conversations early in the review process with those who 
have an interest in the evaluation. Such communication 
will need to occur throughout the evaluation to ensure that 
data being collected relate to original and emerging ques- 
tions and to test the validity of conclusions and recommen- 
dations. This emphasis on flexibility and frequent commu- 
nication has been emphasized repeatedly in the literature 
(Barak 1982; Brown and Braskamp 1980). 

5. In designing an evaluation process, an institution 
should review all programs on a cyclical basis rather 
than selected programs on an ad hoc basis. 

Most institutions reviewed in this study have established 
a review process in which all programs are evaluated cycli- 
cally. This arrangement has the advantage of making pro- 
gram review a routine activity, one in which the advent of 
a review does not raise anxieties prematurely. Those insti- 
tutions that conduct ad hoc reviews in response to prob- 
lems are likely to find themselves faced with the need to 
justify the review continuously and to deal with apprehen- 
sions about the review at every stage of the process. It will 
not take long for program participants to learn that those 
programs designated for review have problems — hardly a 
healthy context in which to carry out ; review. 

6. Very few evaluative questions can be answered with 
any degree of certainty on the basis of a single indicator. 
Most questions have multiple dimensions, and multiple 
indicators will be required for their assessment. 

The indicators to use in making evaluative judgments are 
never straightforward. Opinions range widely about how 
this task should be done, what measures should be used, 
and what the results mean. Similar debates have occurred 
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with respect to assessing a program's demand, centrality, 
uniqueness, efficiency, and effectiveness. Each of these 
concepts includes a number of indicators that one might 
suggest, but it is unlikely that any one will suffice. The 
consensus in the literature is that most evaluative ques- 
tions are best addressed by collecting data from multiple 
sources (Brown and Braskamp 1980; Clark, Hartnett, and 
Baird 1976a, 1976b; DiBiaso et al. 1981; Lawrence and 
Green 1980). This strategy recognizes the difficulty of mak- 
ing evaluative judgments and suggests that they are more 
likely to be accurate if different dimensions are measured 
and different perspectives solicited. 

7. Provided quality is assessed in a multidimensional 
way, the number and variety of quality indicators identi- 
fied to date provide a basis for making reasonable judg- 
ments about the strengths and limitations of programs. 
The several strategies for assessing quality discussed in 

this monograph provide some insight into program quality. 
Considerable attention has been given to assessing faculty 
research, somewhat less to instructional quality. Attempts 
to assess instructional quality often focus on resources 
(success of graduates, for example) and pay little attention 
to program impact. This deficiency should be corrected 
through attention to "value-added" learning (Astin 1980). 
Unfortunately, most program reviews include few mea- 
sures of program impact. 

A great number of quantitative and qualitative indicators 
have been su^ ssted fox use in assessing the quality of a 
program (Clark, 1 4 artnett, and Baird 1976b; Conrad and 
Blackburn In press a; Webster 1981). The specific indica- 
tors to be used and the weight to be assigned to each need 
to vary by institution. If quality is to be adequately 
assessed, however, it is imperative that institutions use 
diverse indicators (both quantitative and qualitative) 
indicators not only of program resources and outcomes but 
also of value-added student learning. 

8. The use of outside reviewers need not be an integral 
part of every program review. 

The use of outside reviewers has gr^* ! support in higher 
education, and their use is justified on Jie basis of the dis- 
ciplinary perspective and wisdom added to the review. 
Although outside reviewers undoubtedly provide valuable 
insight into some matters, it also seems reasonable to 
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assume that disciplinary questions may not always be para- 
mount. Such issues as quality of instruction, effective lead- 
ership, student demand, centrality, and efficiency do not 
automatically require the perspective of outside reviewers. 
Furthermore, such reviews are expensive, both in terms of 
dollars and in terms of time. Outside reviewers should be 
used selectively in response to specific questions that such 
a perspective would help resolve. 

Results 

9. Institutions need to be concerned about the link 
between program review processes and other decision- 
making processes. This link will not occur automatically; 
it must be incorporated into the design of the review pro- 
cess and must be nurtured continuously. 

Most evaluation processes are isolated from decision 
making. This problem is particularly acute at institutions 
where program review is conducted in one part of the insti- 
tution and planning and budgeting in another. A few insti- 
tutions (Ohio State University, for example) have devised 
procedures to ensure that tho results of reviews are intro- 
duced into planning and budgeting. More typically, how- 
ever, the link between evaluation and planning and budget- 
ing does not exist. Evaluations should be linked more sys- 
tematically to planning and budgeting (Arns and Poland 
1980; Clifford and Sherman 1983; Craven 1980b; Hartmark 
1982; Patton 1978; Stufflebeam et al. 1971). 

10. Program review activities should be linked to but not 
drive decision-making processes 

This proposal is related to the previous one, and it 
serves to attenuate expectations to a reasonable level. It is 
important for the results of program review to be linked to 
decision making, but it is unreasonable to expect this link 
to result in a one-to-one relationship between evaluative 
conclusions and recommendations and subsequent actions. 
Just as evaluations rely on multiple indicators, decisions in 
other parts of the institution are improved if multiple per- 
spectives are solicited. Those responsible for program 
reviews should ensure that the results constitute one 
source of information that is routinely consulted. Some- 
times the evaluative advice should be followed; sometimes 
it should be ignored. 

One assumption implicit in program review is that such 
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reviews are likely to make a positive contribution. The fact 
is that research on this matter is almost nonexistent. A few 
institutional case studies have examined whether results 
have been used, but the larger issue — the question of 
impact — has not been examined systematically. On the one 
hand, the continued existence and growth of program 
review processes suggest that such efforts are supported 
and that the results can be beneficial. On the other hand, 
numerous critics believe that such reviews do not make 
much difference and, in fact, may actually make things 
worse. Clearly, researchers and practitioners alike need to 
give increased attention to the assessment of the conse- 
quences of program review. 
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